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Assistant  Secretary  of  Defense  (L&MR)  Alan  F.  Estevez 

The  top  DoD  official  for  logistics  and  materiel  readiness  discusses  a  wide  range  of 
topics,  from  how  we  sustain  the  force  amid  budget  constraints,  to  supporting  the 
troops  still  in  Afghanistan  even  as  we  draw  down  forces  there,  to  how  we 
can  best  prepare  the  next  generation  of  acquisition  professionals. 
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Improving  Army  Systems  Develop¬ 
ment  through  Early,  Experienced  User 
Involvement 

Jeffrey  D.  From  and  Brett  R.  Burland 

The  Army  Mission  Command  Battle 
Lab  and  the  Army  Command  and  Gen¬ 
eral  Staff  College  created  the  Science  & 
Technology  Seminar-Brownbag  Lunch 
Program  as  a  way  to  get  end  users  and 
technology  developers  communicating 
early  in  the  process. 


The  Fallacy  of  Quantifying  Risk 

David  E.  Frick ,  Ph.D. 

in  crafting  a  risk  analysis  for  presenta¬ 
tion  to  a  Milestone  Decision  Authority, 
be  aware  of  the  role  organizational 
culture  can  have  in  influencing  how 
risks  are  presented.  Affixing  labels  of 
low,  medium,  and  high  to  risks  fails  to 
account  for  the  unexpected.  A  full  de¬ 
scription  of  the  known  risks,  along  with 
acknowledgement  of  those  that  lie  hid¬ 
den,  is  crucial  to  an  accurate  analysis. 
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How  to  Form  an  IPT 

David  Hofstadter 

For  an  integrated  product 
team  to  succeed,  both 
the  leader  and  the 
team  members  must 
be  carefully  chosen 
based  on  expertise.  In 
addition,  all  must  commit 
to  attitudes,  practices,  and 
common  understandings 
that  will  allow  the  team  to 
objectively  analyze,  evaluate, 
and  decide. 
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An  Immodest  Proposal 

Making  a  Profit  on  Defense  Programs 
John  Krieger 


Scientific-based  test  design  can  help 
develop  rigorous  test  and  evaluation 
experiments,  particularly  in  the  Plan 
and  Design  phases  of  the  Design 
of  Experiments  approach. 
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Although  the  purpose  of  DoD  orga¬ 
nizations  is  not  to  make  a  profit,  they 
can  still  help  make  ends  meet  through 
recoupment  of  the  value  of  technologies 
and  systems  that  are  adaptable  to  the 
needs  of  industry  and  the  public. 
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The  author  explains  that  accurately 
projecting  sustainment  costs  is  hard— 
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certain  former  New  York  Yankee  (and 
Navy  veteran)  might  say. 
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arlier  this  year,  Assistant  Sec¬ 
retary  of  Defense  for  Logistics 
and  Materiel  Readiness  Alan 
Estevez  spoke  before  a  class 
of  acquisition  professionals  at 
National  Defense  University. 
They  presented  a  number  of 
questions  that  he  agreed  to  dis¬ 
cuss  again  with  the  readers  of 
Defense  AT&L. 

DAT&L:  We  all  are  aware  that  DoD  has  been  tasked  to  do  more  with  less 
and  prepare  for  budget  cuts  in  the  future.  Given  that  how  do  we  keep  from 
becoming  a  hollow  force  in  the  years  ahead? 

Estevez:  Well,  you  really  have  to  go  back  to  how  we  crafted  the  budget. 
The  fact  is,  the  first  thing  we  did  prior  to  developing  a  budget  is,  we  de- 
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vised  a  defense  strategy,  recognizing  the  targets  that  were  in 
the  Budget  Control  Act  for  that  budget.  The  defense  strategy 
came  before  we  put  dollars  against  programs  in  the  budget. 
So  we  built  a  strategy  that  rebalances  our  global  posture  and 
presence,  emphasizing  Asia-Pacific  while  continuing  our  ef¬ 
forts  in  the  Middle  East.  Of  course,  this  doesn't  mean  that  we 
walk  away  from  our  commitments  globally.  After  reshaping 
our  defense  strategy,  we  sized  the  force  and  our  programs 
around  that  strategy,  carefully  crafted  with  the  Services  and 
Service  leadership. 

We  then  built  a  budget  that  can  sustain  the  force  that  we  have 
which  in  turn  sustains  the  strategy.  Now  with  that  said,  we 
must  remain  capable  of  responding  to  the  changing  nature  of 
warfare.  This  will  potentially  require  us  to  make  adjustments 
in  the  budget  to  cover  those  gaps. 

In  the  budget  submission,  we  sustained  R&D,  and  we  essen¬ 
tially  sustained  our  modernization  programs.  We  ensured 
some  of  the  poor  performers  were  terminated.  By  ensuring  a 
deliberate  planning  process  in  developing  the  strategy,  force 
structure,  and  the  budget,  it  demonstrated  that  we  have  the 
ability  to  sustain  the  force  structure  as  designed— which 
should  preclude  DoD  from  having  a  hollow  force. 

DAT&L:  In  addition  to  R&D  and  modernization ,  what  other  priori¬ 
ties  are  there? 

Estevez:  From  a  logistics  standpoint,  I  essentially  have  four 
priorities:  First,  sustaining  current  operations.  We  are  still  en¬ 
gaged  in  Afghanistan,  which  is  a  very  tough  place  from  the 
logistics  standpoint  to  be  at  war,  because  it's  a  landlocked 
country.  And  it's  surrounded  by  at  least  one  country  that  is 
definitely  not  our  friend  and  other  countries  that  have  their 
own  ways  of  doing  things.  And  so  we  have  to  work  through 
that.  So  doing  that  is  job  one.  And  there  are  a  couple  of  facets 
to  that:  One  is  again,  sustaining  the  forces  on  the  ground- 
continuing,  as  the  enemy  adapts,  to  adapt  back. 

So  under  the  leadership  of  Sec.  Panetta,  Dr.  Carter,  and  previ¬ 
ously,  Sec.  Gates,  we  put  a  great  effort  into  increasing  the  ca¬ 
pability  of  the  force— ISR  capabilities,  small  devices  like  hand¬ 
held  ground-wire  detectors  that  find  lEDs,  and  MRAPS.  There 
are  about  14,000  MRAPS  of  varying  types  in  Afghanistan,  and 
we're  sustaining  those  at  a  well  over  90  percent  readiness 
rate— very  good. 

Even  with  closure  of  PAK  GLOC  [Pakistani  Ground  Lines  of 
Communication],  we've  done  a  great  job  of  sustaining  the  force 
there.  We  had  some  challenges  with  food  and  fuel,  but  essen¬ 
tially,  we're  actually  on  the  rise  for  both  of  those  commodities 
right  now.  And  that,  frankly,  is  because  of  the  great  efforts  of 
the  logisticians  in  theater,  the  Defense  Logistics  Agency,  and 
United  States  Transportation  Command  in  supporting  that. 


We  can  talk  about  retrograde  down  the  pike:  Getting  out  of 
Afghanistan  is  also  going  to  be  a  real  trick— much  tougher  than 
the  retrograde  from  Iraq,  which  was  no  mean  feat  in  and  of 
itself.  So  that's  job  one. 

Moving  down  the  line:  We've  learned  many  lessons  in  the  last 
10  years  or  so  in  our  contracting  environment— contracting  for 
support  on  the  battlefield.  We  have  a  couple  of  lessons  that 
were  provided  by  other  organizations.  Some  of  those  recom¬ 
mendations  we  were  learning  ourselves.  We  have  the  Gansler 
Commission  report.  We  have  the  Commission  on  Wartime 
Contracting  report.  And  I  would  say  that  we've  done  a  great 
job  in  the  last  5  years  of  turning  ourselves  around  from  where 
we  were  in  contracting  on  the  battlefield  and  managing  the 
contractors  that  are  on  the  battlefield.  Developing  whole  op¬ 
erational  contract  support  construct  continues  to  be  one  of 
my  priorities. 

Now  the  challenge  is  that  we've  done  that  for  the  near  term. 
The  way  I  like  to  term  that  is:  We  had  a  gaping  wound.  We've 
sewn  that  up  and  stanched  the  bleeding.  But  really,  the  key 
to  success  is  to  embed  that  in  the  DNA  of  the  culture  going 
forward.  So  it's  not  only  about  this  war  now,  it's  also  about 
having  a  plan  for  future  contingencies.  And  there's  still  some 
work  to  be  done  there. 

Priority  3  is  life  cycle  logistics  management.  How  do  we  embed 
in  the  thought  process  that  we  need  to  sustain  what  we  are 
buying  new  for  the  next  30,  40,  50  years,  depending  on  the 
platform  that  we're  buying?  That's  pretty  important.  Thirty 
percent  of  the  cost  of  a  platform  is  in  its  research  and  de¬ 
velopment  acquisition,  while  the  remaining  70  percent  is  in 
sustaining  that  [product]  over  its  life  cycle.  It  becomes  pretty 
important  to  buy  a  weapon  system  so  that  it  has  a  lower  cost 
in  sustainment.  We've  done  a  lot  to  raise  the  level  of  that  dis¬ 
cussion  at  defense  acquisition  boards  and  within  the  defense 
acquisition  community. 

Equally  important  to  the  mix  is  the  development  of  solid  sus¬ 
tainment  strategies.  Once  you've  bought  it,  how  are  you  going 
to  sustain  it?  There  are  a  number  of  different  ways  of  doing 
that,  including  performance-based  logistics  [PBL]  strategies. 
How  do  you  do  these  PBL  strategies  well?  We  have  a  whole 
area  of  assessments  around  that  we  are  working  with  the  ac¬ 
quisition  community  and  the  sustainers  of  that  community,  to 
ensure  that  that's  embedded  in  their  thought  process. 

The  final  priority  is  what  I'll  call  excellence  in  logistics/supply 
chain.  How  do  we  do  our  business  well?  How  do  we  increase 
the  capability  of  our  tradecraft  in  that  area?  Improving  our 
processes  increases  military  capability.  If  you  are  sustaining 
a  force  on  the  battlefield  well,  you  are  freeing  up  capability 
to  do  other  things.  Although  we  didn't  start  off  as  well  as  de¬ 
sired,  we've  performed  exceptionally  over  the  last  number  of 
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years  sustaining  and  moving  the  force  in  Afghanistan  and  Iraq. 
Nonetheless,  there  are  things  we  could  do  better. 

And  then  just  in  the  general  industrial  base  side  of  it,  the  nor¬ 
mal  business  of  the  Department— again,  there  are  processes 
we  can  do  better  to  both  increase  our  capability  and  lower  our 
costs.  Lowering  our  costs  is  important  in  this  environment. 
If  I  can  take  cost  out  of  the  logistics  business  area  without 
decreasing  capability,  I'm  freeing  up  dollars  for  other  require¬ 
ments  inside  the  Department.  I  think  that's  important. 

DAT&L:  Can  you  discuss  opportunities  DoD  is  leveraging  to  share 
technology  and  resources  in  new  ways? 


Years  Defense  Program]  return  on  investment— just  by  man¬ 
aging  the  distribution  process  and  the  network  around  that 
better.  And  I'd  say  that's  a  down  payment;  there  are  more 
efficiencies  to  be  gained  in  that  area. 

DAT&L:  Can  we  use  working  capital  funds  for  reliability  upgrades? 

Estevez:  Depends  on  how  you  bought  the  platform  and  how 
you're  sustaining  that.  So  if  a  working  capital  fund  happens  to 
be  paying  the  performance-based  logistics  on  a  contract  and 
the  PBL  contract  on  component  X  includes  reliability  upgrades, 
sure.  But  in  general,  reliability  upgrades  tend  to  be  paid  out  of 
procurement  accounts. 


We're  leveraging,  of  course,  DLA.  We  just 
did  an  assessment  called  strategic 
network  optimization,  which 
has  the  potential  to  provide 
DoD  with  a  couple  of 
hundred  million 
dollars  over  the 
FYDP  [Future 


The  key  to  success 
is  to  embed  that  in  the  DNA 
of  the  culture  going  forward. 

So  itrs  not  only  about  this  war  now, 
jit's  also  about having  a  plan  for 


^uture  contingencies.  And  there's 
still  some  work  to  be  done  there. 


Estevez:  Joint  strike  fighter  is  obviously  a  coalition  platform, 
but  there's  some  work  to  be  done  there.  We're  looking  at  joint 
capabilities  inside  the  Department.  For  example,  when  we  look 
at  our  depot  structure,  how  do  we  optimize  our  depots  to  en¬ 
sure  that  they're  getting  the  right  workload  at  the  right  depots 
across  the  joint  community,  versus  Service-specific?  We  just 
did  a  major  assessment  in  the  UAV  area,  where  we're  targeting 
specific  depots  to  do  that  workload.  So  instead  of  scattershot- 
ting  that  capability  (because  every  Service  has  its  own  UAV 
capability  because  of  mission  sets),  if  we  develop  capability  in 
Depot  X,  why  can't  Depot  X  do  that  for  the  joint  community 
versus  one  particular  Service?  That'll  give  us  some  savings. 


DAT&L:  How  do  PBL  contracts  impact  Service  force  structure 
and  design? 


Estevez:  There's  a  range  of  how  PBL  contracts  operate.  We 
just  did  a  study  called  ProofPoint  [ Editor's  note:  See  the  March- 
April  2012  issue  for  an  article  about  this  study.]—  two  iterations 
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of  it— that  did  deep  dives  into  a  number  of  programs.  Some 
of  them  show  real  benefit;  others  had  some  problems.  But  in 
general,  if  you  look  across  the  continuum  of  contracts,  a  well- 
done  PBL  contract  can  save  dollars  or  increase  readiness  or, 
in  the  best  of  cases,  both. 


we  certainly  could  say  that  what  we  have  done  will  increase 
the  readiness  while  decreasing  the  capability  needs  to  sustain 
that  readiness  against  a  particular  platform. 

DAT&L:  Should  OSD  drive  the  alternate-fuel  development  or  let 
the  commercial  sector  drive  and  we  follow? 


Now  with  that  said,  they're  not  the  easiest  contracts  to  develop. 
So  you  really  need  A-team  contracting  officers,  A-team  sustain¬ 
ed,  and  A-team  program  managers  to  develop  the  right  con¬ 
tract  construct  for  what  it  is  you're  putting  under  a  PBL  contract. 

I'm  not  sure  a  PBL  contract  changes  force  structure,  though  it 
could  change  how  many  people  you  need  sustaining  an  item 
out  there  on  the  battlefield— or  back  in  depot,  for  that  matter. 
I  don't  know  that  we've  changed  how  many  X  platforms  as  a 
whole  that  we  buy.  For  example— if  we  buy  joint  strike  fighters, 
we  buy  against  a  threat  and  a  need.  Theoretically,  if  I  could 
really  guarantee  X  percentage  of  readiness,  I  would  buy  less.  I 
don't  think  we're  quite  there  yet  in  changingthe  dynamics,  but 


Estevez:  Well,  let's  start  out  [noting]  that  industry  is  who  is 
out  there  developing  alternative  fuels,  not  the  Department.  It's 
not  work  that's  being  done  in  our  labs;  it's  being  done  in  the 
commercial  sector.  The  Department  definitely  has  a  role,  as 
long  as  there's  a  link  to  military  value  of  putting  dollars  against 
capabilities— in  this  case  alternative  fuel— that  can  help  jump- 
start  that  tool  [or]  technique. 

The  general  thought  process  on  alternative  fuels  would  be  that 
they  would  increase  the  overall  supply  of  petroleum  product. 
That  does  two  things:  It  can  lower  the  cost  of  the  product. 
Petroleum  is  based  on  a  worldwide  marketplace  and  is  a  fun¬ 
gible  product.  It  could  [also]  increase  U.S.  security.  Both  those 
things  are  in  the  Department's  interest.  So  the  Department 
putting  some  dollars  against  alternative-fuel  capability  makes 
perfect  sense. 

DAT&L:  How  does  OSD  balance  modernization  with  reset  using 
overseas  contingency  operations  [OCO]  funds  (or  otherwise)? 


roving  our  processes 
increasSsinilitary  capability.  If 
you  are  sustaining  a  force  on  ffte 


battlefield  well,  you  ar^  freeing 
up  capability  to  do  other  things. 
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Estevez:  The  basic  rules  say  that  OCO  is  not  to  pay  for  mod¬ 
ernization;  investment  accounts  are  to  pay  for  moderniza¬ 
tion.  With  that  said,  there  are  upgrades  that  are  going  on  the 
battlefield  for  needs  right  now,  today,  that  OCO  is  paying  for. 
So  if  I  upgrade  an  MRAP  with  an  underbelly  kit,  increasing  its 
capability  to  sustain  a  blast  and  protect  its  riders— that  would 
be  viewed  as  modernization.  That  is  paid  for  out  of  OCO.  But 
if  I'm  going  to  do  Apache  Block  III,  that  portion  of  it  would  be 
paid  out  of  investment  accounts. 

DAT&L:  Will  OCO  budgets  be  rolled  into  O&M? 

Estevez:  Well,  OCO  is  not  just  O&M.  There  is  OCO  that  pays 
for  investment,  OCO  that  pays  for  R&D—  JIEDDO  [Joint  Im¬ 
provised  Explosive  Device  Defeat  Organization]  work,  for  ex¬ 
ample— that  turns  into  a  real  capability  back  on  the  battlefield 
today,  which  is  what  OCO  is  designed  to  do— to  sustain  the 
fight  that  we're  in. 

Obviously,  there  are  pressures  on  budget  as  a  whole,  includ¬ 
ing  pressures  on  the  OCO  budget.  And  as  we  complete  our 
mission  in  support  of  contingency  operations,  OCO  budgets 
will  decline— not  necessarily— and  hopefully  not,  in  fact— as 
a  one-to-one.  Because  there's  a  requirement  after  we  draw 
down  in  Afghanistan— I  can't  predict  what  the  force  structure 
in  Afghanistan  will  be  post-2014.  We  recently  signed  a  security 
agreement  with  the  Afghans  that  calls  for  a  U.S.  and  Afghan 
partnership  post  the  2014  drawdown  of  combat  operations  in 
Afghanistan.  And  how  that  will  be  paid  for— that's  a  prediction 
I'm  not  prepared  to  make. 

But  I  do  know  that  as  equipment  comes  out  of  Afghanistan, 
it  will  need  to  be  reset.  Equipment  doesn't  exactly  come  out 
as  the  force  comes  out,  and  our  hope  is  that  there  are  OCO 
funds  available  to  pay  for  that  reset.  We  all  know  that  there's 
a  bill  to  be  paid  after  the  fact.  Again,  I  cannot  predict  how 
the  budgets  will  transpire  in  Congress  during  the  2014,  2015 
timeframe. 

DAT&L:  What  is  the  vision  of  logistics  and  our  posture  in  the 
Pacific?  Are  there  any  specific  initiatives? 

Estevez:  A  couple  of  things  I  will  say:  The  Pacific  is  pretty  large. 
That  makes  logistics:  1)  important  and  2)  harder  to  manage. 
Now  again,  we  just  did  logistics  in  two  wars,  one  of  which  was 
in  a  landlocked  country.  Most  people  forget  that  based  on  the 
president's  decision  in  February  2009,  we  deployed  a  couple 
of  brigades  (20,000  people)  throughout  that  year.  Addition¬ 
ally,  when  the  president  said  to  surge  forces  in  Afghanistan 
in  December  2009— increase  the  force  by  30,000— U.S. 
TRANSCOM  and  CENTCOM  worked  through  the  numbers. 
TRANSCOM  essentially  said,  "You  can't  fit  any  more  equip¬ 
ment  or  materiel  into  the  flow  pattern  in  order  to  close  that 


force  by  August  of  2010.''  Yet  we  managed  to  put  in  7,000-plus 
MATVs  into  that  flow  pattern. 

So  logistics  is  capable  of  incredible  agility  and  flexibility.  And 
that's  again  because  of  the  great  logisticians  out  there  on  the 
battlefield,  the  great  work  of  TRANSCOM,  the  military  Ser¬ 
vices,  and  the  Defense  Logistics  Agency,  in  doing  that. 

So  in  talking  about  the  Pacific:  We  do  have  a  logistics  laydown 
in  the  Pacific;  it's  not  like  we  are  just  suddenly  going  there. 
Obviously,  we  have  forces  in  Korea  today.  We  have  forces  in 
Japan  today.  One  thing  that  will  happen  is  our  forces— cer¬ 
tainly  Marine  Corps  forces— will  be  more  dispersed.  We'll  have 
to  look  at  that  dispersion  in  relation  to  our  mobility  require¬ 
ments.  We're  about  to  do  a  new  mobility-requirements  as¬ 
sessment,  based  on  the  new  force  structure.  The  last  one  was 
done  on  a  force  structure  that  had  the  Army  and  the  Marine 
Corps  at  100,000  more  people  than  they  will  have  in  the  future 
force  structure.  So  that  changes  the  mobility  requirements 
right  there. 

But  increasing  dispersion  of  the  force  structure  will  require 
some  mobility  requirements  to  cover  that  slack.  Between  our 
force  of  C-17s,  modernized  C-5s  (and  we're  increasing  the  ca¬ 
pability  of  the  C-5B  fleet,  being  changed  into  C-5Ms)— also 
our  C130s  and  our  CRAF  [Civil  Reserve  Air  Fleet]— our  con¬ 
tingency  airlift  fleet,  which  is  commercial  carriers  that  we  rely 
on— my  expectation  is  that  given  the  numbers,  we'll  be  able  to 
sustain  the  force  laydown  that  we  have  in  the  Pacific,  but  we'll 
see  what  happens  as  we  do  this  study. 

DAT&L:  Do  you  have  any  thoughts  about  the  relationship  ofDLA 
and  TRANSCOM  and  possible  merger? 

Estevez:  They  have  a  great  relationship!  First  of  all,  if  you're 
goingtotalkaboutthis,  you  probably  cannot  limit  it  to  TRANS¬ 
COM  and  DLA;  they  are  two  components  of  the  logistics 
structure.  Merging  the  relationship  of  those  two  really  is  a 
piece— and  I  don't  think  it's  the  biggest  piece,  frankly— of  the 
logistics  structure.  I  think  that  the  relationship  of  TRANSCOM 
and  DLA  is  great.  TRANSCOM  and  DLA  have  different  mis¬ 
sions.  They  have  different  focuses.  They  work  very  well  to¬ 
gether  in  sustaining  the  force  on  the  battlefield. 

If  you  start  looking  at  DLA:  DLA  probably  buys  about  80  per¬ 
cent  of  the  materiel  for  the  Department— I'm  talking  about 
repair  parts.  It  increases  when  you  include  food  and  fuel,  which 
they  are  able  to  provide  in  a  lot  of  different  places.  Food  and 
fuel  are  provided  all  over  the  world.  However,  it  is  important 
to  note  that  while  DLA  buys  about  80  percent  of  our  repair 
parts,  the  military  Services  buy  the  remaining  20  percent;  that 
20  percent  is  actually  80  percent  of  the  value  of  the  inventory, 
which  for  the  most  part,  are  readiness  drivers.  So  they  are  as 
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much  in  the  mix  as  DLA  and  TRANSCOM  when  you  start 
talking  about  sustaining  the  force  on  the  battlefield. 

And  here  again,  we  are  showing  that  we  do  it.  DLA  is  all 
over  the  battlefield  today,  doing  distribution  capability,  doing 
food,  fuel,  support  capability,  doing  disposition.  DLA  cuts 
up  battle-damaged  equipment,  turns  it  into  scrap.  All  that 
is  being  done  on  the  battlefield  today  by  DLA.  None  of  what 
I  just  talked  about,  except  for  the  distribution  piece,  is  in 
TRANSCOM's  mission  set.  And  I  don't  think  TRANSCOM's 
looking  for  that  mission  set.  And  when  you  really  look  at  DLA, 
it's  about  a  $42-billion-a-year  buying  agency  with  about  $3 
billion  or  $4  billion  of  distribution  depot  operations  rolling 
in  on  top  of  that. 

DAT&L:  Speaking  of  depots:  One  NDU  student  asks:  If  I  put  a 
dollar  into  a  depot ,  how  do  I  get  a  dollar  of  readiness? 


Estevez:  First,  let  me  just  say  our  depots  do  some  great  work. 
We  are  probably  over  capacity  in  the  depot  structure.  We  will 
be  reviewing  that,  and  if  Congress  authorizes  a  BRAC,  we'll 
definitely  be  reviewing  depot  structure.  But  the  depots  do  a 
great  job.  If  you  put  a  piece  of  equipment  into  the  depot,  you 
are  going  to  get  a  great  piece  of  equipment  out  on  the  other 
end  of  that.  When  you  start  looking  at  depots,  you  start  look¬ 
ing  at  the  overhead  structure  behind  them;  there's  a  cost  to 
doing  that.  From  a  pure  touch-labor  standpoint,  depots  are  as 
competitive  as  any  commercial  entity  out  there  in  doing  that. 

So  when  I  talked  earlier  about  supply-chain  excellence,  you 
have  to  look  at  it  in  the  total.  It's  not  just  the  depot  opera¬ 
tions.  You  have  to  look  at  the  logistics  system  that  sustains 
that  depot,  both  the  commercial  industry  piece  of  that  and 
the  organic  piece  of  that,  to  determine  whether  you're  achiev¬ 
ing  value.  In  general,  I'd  say  yes.  But  could  we  do  better?  The 
answer  is  also  yes. 


If  we're  going  to  operate  in  a 

*  4 

constrained  resource  environments  - 
having  the  appropriate  tradecraft  and 
the  skill  set  is  the  only  way  we  are 
going  to  achieve  great  support. 


Now  let  me  just  add  another  thought:  When  you  start 
talking  about  this  mix— in  our  earlier  discus¬ 
sion  about  PBL,  many  of  those  are 
done  under  depot  partnership 
arrangements.  But  it's 
also  important,  as 
we  compress 
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spending  inside  the  Department  based  on  our  budget  require¬ 
ments,  that  we  sustain  a  vibrant  industrial  base.  Dr.  Carter 
has  been  quoted  as  saying  that  when  we  decide  to  buy  a  new 
airplane,  a  new  combat  vehicle,  a  new  truck,  a  new  ship,  we 
go  to  the  commercial  sector  to  do  that.  We  do  not  have  that 
capability  inside  the  Department  of  Defense.  So  it's  important 
to  have  those  people  out  there  when  it  comes  time  to  do  that. 
And  it's  not  just  the  company,  who  you  might  go  to  in  order  to 
buy  that  equipment.  It's  the  second-,  third-,  fourth-tier  suppli¬ 
ers  we  also  rely  on,  on  the  sustainment  side. 

So  it  can't  be  all  about  the  organic  structure.  It  has  to  be  about 
the  industrial  base  and  the  organic  structure  at  the  optimum 
mix  to  achieve  optimum  results  of  readiness  and  optimum  re¬ 
sults  of  investment,  through  procurement,  for  the  dollar  spent 
by  the  Department  of  Defense. 

DAT&L:  Another  question  is  about  the  Afghanistan  drawdown , 
which  you  discussed  earlier. 

Estevez:  Let  me  just  give  you  a  couple  of  thoughts  about  why 
Afghanistan  is  going  to  be  so  much  more  challenging  than 
Iraq.  In  Iraq,  we  had  Kuwait  there  as  a  great  base  of  operations. 
Kuwait  has  great  port  facilities.  The  Kuwaiti  government  let 
us  operate  [Army  Camp]  Arifjan  on  their  soil  and  some  other 
sites,  so  that  we  were  able  to  bring  forces  in,  set  them  up,  get 
them  trained,  send  them  north— the  same  capability  was  there 
in  turning  that  around  to  redeploy  back.  Plus  we  had  logistics 
forces  in  Kuwait  that  were  able  to  do  a  big  piece  of  the  draw¬ 
down.  We  do  not  have  that  same  capability  in  Afghanistan. 
There  is  no  Kuwait  as  a  "catcher's  mitt,"  where  you  can  move 
all  this  equipment  to  and  do  wash  racks  and  prep  capability 
before  you  move  it  back  to  the  States. 

Afghanistan  is  a  landlocked  country,  and  it  has  a  road  network 
that  is  nowhere  near  the  road  network  of  Iraq.  We're  still  in 
contact  with  the  enemy  in  Afghanistan.  Right  now  we  have 
one  or  more  main  routes  into  Afghanistan  shut  down  in  both 
directions.  That  would  be  our  main  route  for  ground  movement 
out  through  Pakistan.  My  expectation  is  routes  will  open  up  at 
some  point— not  entirely  sure  when.[  Note:  Pakistan  re-opened 
these  routes  in  July  2012.]  Northern  distribution  routes,  which 
have  been  fabulous  for  our  sustainment  mission  in  Afghani¬ 
stan— we  really  haven't  exercised  them  for  retrograde.  We 
are  starting  to  do  that.  It  will  take  a  while  before  we  hone  the 
practices  of  that;  this  includes  multiple  border  crossings  of  the 
Central  Asian  states,  plus  Russia,  plus  the  Baltic  States.  It  will 
take  a  while  before  we  get  flow  really  going  either  through  the 
northern  distribution  network  or  through  Pakistan,  whenever 
Pakistan  and  we  reach  an  agreement  to  reopen  the  routes 
through  Pakistan. 

And  of  course,  we  are  flying  equipment  out.  We  can  do  that 
with  all  those  multimodal  capabilities,  and  TRANSCOM  is  run¬ 


ning  some  operations  to  do  that.  But  all  in  all,  it's  going  to  be 
a  pretty  difficult  operation.  And  it's  not  about  moving  people; 
it's  about  moving  equipment.  And  moving  people  is  also  more 
complex  in  Afghanistan  than  it  was  moving  them  out  of  Iraq. 

DAT&L:  Is  someone  looking  ahead  at  any  " leapfrog "  technology 
for  OSD  and  ERPs? 

Estevez:  The  question  is  pretty  broad.  I'll  hone  it  back  to  the 
IT  portion  of  that.  Obviously,  our  R&D  structure  and  the  Ser¬ 
vice  labs  are  certainly  looking  at  next  generation  technology. 
And  DARPA— its  business  is  looking  at  next-gen  technology. 
And  obviously,  we  are  going  to  start  looking  at  the  grid  and 
the  infrastructure  on  that,  how  to  utilize  cloud,  and  where  the 
commercial  sector  is  advancing— all  that's  going  to  play  into 
the  Department's  strategy. 

From  an  ERP  [enterprise  resource  planning]  perspective  and 
an  IT  perspective,  the  Department  does  not  do  IT  well.  I  wish 
we  would  do  it  better.  Each  of  the  Services  has  made  some 
major  investments  in  ERPs  over  the  last  10  years.  Most  of  those 
are  going  to  come  into  fruition  in  the  next  couple  of  years. 
Some  of  them  are  already  operating. 

We  need  that  to  happen  for  a  couple  of  reasons:  One,  for  all 
those  things  I  talked  about  in  supply  chain  excellence  and 
lowering  costs:  ERPs  give  you  great  capability  to  look  holisti¬ 
cally  in  that  regard.  And  I  think  we're  going  to  see  the  benefits 
of  that  in  the  coming  years.  So  it's  not  just  about  ERPs;  it's 
about  the  business  process  that  you  put  in  behind  that  as  well. 
Some  Services  are  not  as  advanced  as  others,  and  there  will 
be  problems. 

Second— and  we  do  have  a  major  focus  in  the  Department 
given  that  the  Secretary  has  emphasized  this  and  actually  put 
goals  on  it— is  auditability.  We  do  have  a  legal  requirement  to 
be  auditable.  The  government  as  a  whole— the  Department 
is  going  to  be  the  last  Cabinet  Department  that  has  a  clean 
audit  statement.  ERPs  are  a  foundation  to  help  the  Depart¬ 
ment  have  a  clean  audit  and  track  equipment  on  the  books; 
frankly,  I  think  the  American  taxpayer  expects  us  to  be  able 
to  track  our  books. 

DAT&L:  One  DAU  dean  asks:  OSD  Log  was  at  the  human  capital 
forefront  a  few  years  ago  by  issuing  the  DoD  Logistics  Human 
Capital  Strategy  in  2008.  What  have  we  learned  about  leading 
our  talent and  what  can  DAU  and  our  stakeholders  do  to  help 
further  this  initiative? 

Estevez:  I'll  talk  about  log  as  a  piece  of  that.  Human  capital  is 
critical,  obviously,  across  the  Department— not  just  in  the  log 
area,  but  you  can  name  an  area,  and  there  are  little  fiefdoms 
in  some  of  these  areas  of  the  acquisition  workforce.  And  we 
have  a  major  emphasis  on  growing  and  increasing  the  skillset 
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of  the  acquisition  workforce— something  that  DAU  is  in  the 
forefront  of,  on  the  acquisition  side,  of  course.  A  sliver  of  the 
log  workforce  is  acquisition  workforce;  the  bulk  is  not. 

Cyber  is  a  growing  area.  Medical  capability— the  Department's 
done  some  great  work  in  the  medical  capability  for  the  forces 
that  are  coming  back  that  require  that.  I  had  an  opportunity 
to  visit  Walter  Reed,  and  observed  the  incredible  work  that 
we  are  doing  to  help  our  wounded  warriors  that  are  coming 
back.  That  will  actually  come  out  into  the  public  sector  as  we 
grow  those  capabilities. 

From  the  log  workforce,  what  we  did  a  couple  of  years  ago  is 
we  asked,  what  are  the  skillsets  needed?  Then  we  stood  up 
at  NDU  the  Center  for  Joint  and  Strategic  Logistics— that  Lt. 
Gen.  (retired)  Chris  Christianson  is  heading  up  for  us.  Chris  has 
gone  out  and  looked  at  a  number  of  different  courseware  in 
the  public  university  sectors  and  what  we  teach  in  the  military 
university  system  and  has  identified  some  training  capabilities 
that  we  need  to  increase. 

So  it's  a  mixed  area  on  how  you  do  that— how  you  build  the 
right  logistician.  So  there's  great  university  capability  out  there. 
We  send  people  to  the  universities  I  mentioned  to  ensure  that 
the  right  skillset  is  developed.  It  is  also  critical  that  we  make 
sure  that  our  military  courseware  keeps  pace  with  the  com¬ 
mercial  sector. 

DAT&L:  You  mentioned  areas  where  you  think  there  could  be 
more  logistics  training ,  that  Lt  Gen.  Christianson  is  looking  into— 

Estevez:  Well,  he's  looking  at  a  couple  of  things  in  both  the 
supply  chain  area  and  the  joint  military  education  area.  One 
that  he  likes  to  talk  about  is  assigning  a  junior  major  as  a  plan¬ 
ner  on  the  joint  staff  with  no  joint  experience.  So  what  has 
been  done  to  build  that  officer?  And  that's  some  of  the  areas 


that  he's  looking  at.  I  will  always  raise  the  point  that  you  need 
to  do  that  in  the  civilian  workforce  as  well. 

Then  I  go  back  to  the  question  of:  How  do  I  build  someone  who 
understands  what  good  supply  chain  management  looks  like? 
So  if  I'm  going  to  make  our  depots  and  our  supply  chain  that 
sustains  those  depots  into  a  cost-effective  operation,  as  well 
as  an  effective  operation,  we  need  to  fix  those  capabilities. 
There's  understanding  how  commercial  supply  chain  works; 
there's  understanding  how  the  industrial  base  operates. 

It's  both  experience  and  education  that  get  people  to  under¬ 
stand  that  capability,  so  that  when  I  have  a  supply  planner  at 
Defense  Logistics  Agency,  he  really  needs  to  understand  how 
that  commercial  supply  chain  operates  that  he's  buying  from,  if 
he's  going  to  buy  well.  If  I'm  going  to  have  that  sustainment  op¬ 
erator  who's  going  to  plan  how  I'm  going  to  sustain  joint  strike 
fighter,  or  LCS  or  JLTV  or  ground  combat  vehicle— it's  not  just 
how  does  Boeing  or  Lockheed  or  BAE  or  Northrop  do  it.  It's 
understanding  their  supply  chain.  How  does  that  operate?  So 
having  that  background  and  knowledge  is  value-added  when 
developing  sustainment  contracts  or  framing  a  performance 
based  logistics  contract.  It  is  critical  that  we  understand  where 
we  have  the  best  leverage  and  how  we  can  strike  the  best 
deal.  The  mix  of  skills  that  I  just  went  through:  Contracting, 
acquisition,  logistics  all  need  to  be  in  play  there.  If  we're  going 
to  operate  in  a  constrained-resource  environment,  having  the 
appropriate  tradecraft  and  the  skillset  is  the  only  way  we  are 
going  to  achieve  great  support.  I  think  it  is  important  to  focus 
on  that. 

DAT&L:  Mr.  Estevez ,  thank  you  very  much. 

Estevez:  My  pleasure.  & 

Special  thanks  to  Vice  Director  George  Topic  of  the  Center  for  Joint  and  Stra¬ 
tegic  Logistics ,  National  Defense  University,  Ft.  McNair,  Washington,  D.C. 


MDAP/MAIS  Program  Manager  Changes 


With  the  assistance  of  the  Office  of  the  Secretary  of  Defense, 
Defense  AT&L  magazine  will,  beginning  this  issue,  publish  the 
names  of  incoming  and  outgoing  program  managers  for  major 
defense  acquisition  programs  (MDAPs)  and  major  automated 
information  system  (MAIS)  programs.  This  announcement 
will  list  all  such  changes  of  leadership,  for  both  civilian  and 
military  program  managers. 

For  April- June  2012 

Capt.  Donald  R.  Harder  (USN)  relieved  Capt.  Steve  J. 

McPhillips  (USN)  as  program  manager  for  the  Command  and 
Control  Program  (PMW 150)  on  May  16, 2012. 


Capt.  William  S.  Dillon  (USN)  relieved  Capt.  Michael  T. 
Moran  (USN)  as  program  manager  for  the  Maritime  Patrol 
and  Reconnaissance  Aircraft  Program  (PMA  290)  on  May 
29,  2012. 

Capt.  John  S.  Lemmon  (USN)  relieved  Capt.  Shane  G. 
Gahagan  (USN)  as  program  manager  for  the  Hawkeye,  Ad¬ 
vanced  Hawkeye,  and  Greyhound  Program  (PMA  231)  on  May 
29, 2012.  Gi 
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Our  Troops  Need  Your 

BRAINPOWER 

Here’s  a  way  to  put  it  to  work 


Join  the  best  minds  in  science  and  technology  on  DoDTechipedia— the  new  internal  wiki  for  the 
U.S.  Department  of  Defense.  Post  ideas,  ask  questions,  make  suggestions  or  share  information 
with  colleagues  you  can’t  reach  now.  It’s  a  way  to  expand  our  brainpower  focused  on  rapidly 
responding  to  the  needs  of  the  warfighter. 

HERE’S  HOW  IT  WORKS 

•  Share  your  knowledge.  Every  contribution  counts.  The  more 
you  contribute,  the  more  the  collective  knowledge  base  ex¬ 
pands.  The  wiki  can  easily  be  edited  by  any  user,  broadening 
your  access  to  the  latest  and  best  research  and  ideas.  DoD¬ 
Techipedia  is  open  to  federal  government  employees  and 
contractors  with  Common  Access  Card  or  DTIC  registration. 

•  Connect  across  walls.  Reach  across  command  chains  and 
departmental  divisions  to  find  other  people  working  on 
ideas  and  solutions  that  interest  you.  Discuss  hot  topics. 

Stay  on  top  of  new  trends.  Read  technical  blogs— or  create 
one  of  your  own.  You  don’t  need  to  know  the  right  people— 
you  can  connect  on  the  wiki. 

•  Collaborate.  The  wars  we  are  fighting  today  require  immedi¬ 
ate  solutions.  The  wiki  is  the  biggest  brainstorming  session 
ever  at  DoD.  Network  with  others  working  in  your  areas  of 
interest.  Present  new  ideas  or  technical  challenges.  Stay 
abreast  of  research  and  development  initiatives,  confer¬ 
ences,  and  symposiums.  Collaboration  across  DoD  increases 
our  ability  to  identify  challenges  as  they  emerge  and  deliver 
vigorous  solutions  fast. 


https:yywww.DoDTechipedia.mil 


The  U.S.  Department  of  Defense  Science  and  Technology  Wiki 

A  project  of  Acquisition,  Technology  and  Logistics,  Defense  Research  and  Engineering,  Defense  Technical  In¬ 
formation  Center,  Networks  and  Information  Integration  and  Department  of  Defense  Chief  Information  Officer, 
and  Rapid  Reaction  Technology  Office 


IATAC  MAINTAINS  THE  FOLLOW¬ 
ING  TECHNOLOGY  FOCUS  AREAS: 

Information  Assurance: 

protection  and  defense  of  information  and  IT 

systems 

Information  Warfare: 

capabilities  used  to  exploit  information  and 
IT  systems 

Networking  Technology: 

technologies  that  interconnect  groups  and/ 

or  systems 

IATAC  POC:  Rogelio  Raymond 
(703)984-0072  or 
raymond_rogelio@bah.com 


START  CONTRIBUTING 
TO  DODTECHIPEDIA  NOW 

If  you  have  CAC  or  DTIC  registration, 
you  already  have  access  to  the  wiki. 
Go  to  https://www.DoDTechipedia.mil 
and  log  in.  Once  on  the  wiki,  visit  the 
tutorials  link  to  learn  how  to  add  or 
edit  information. 


Mission  Command 
Battle  Lab  Science  and 
Technology  Seminar 

with  the 

Command  and  General 
Staff  College 


Improving  Army  Systems 
Development  through  Early, 
Experienced  User  Involvement 


Jeffrey  D.  From  ■  Brett  R.  Burland 


The  U.S.  Army  has  long  implored  acquisition  per¬ 
sonnel  to  involve  users  early  and  often  in  the 
process.  An  annual  event  held  by  two  Army 
organizations  has  been  successful  in  bringing 
technology  developers  and  the  potential  users 
together  to  exchange  ideas. 

The  Army  Mission  Command  Battle  Lab  (MCBL)  and  the  Army  Com¬ 
mand  and  General  Staff  College  (CGSC),  both  in  Ft.  Leavenworth,  Kan., 
created  the  Science  &  Technology  Seminar-Brownbag  Lunch  Program 
in  2010  to  expose  CGSC  students,  faculty,  and  staff  to  mission  com¬ 
mand-related  R&D  systems  and  products,  while  giving  developers 
feedback  on  their  work.  The  2-day  seminars  includes  static  displays 
and  demonstrations  on  day  one  and  a  formal  brown-bag  lunch  session 
and  demonstration  on  day  two.  The  schedule  is  closely  coordinated 
with  the  CGSC  leadership  to  ensure  student  availability  and  maximum 


From,  a  retired  Army  lieutenant  colonel,  is  a  science  and  technology  specialist  for  the 
Mission  Command  Battle  Lab  (MCBL)  Capabilities  Development  Integration  Director¬ 
ate  (CID)  at  the  Combined  Arms  Center  in  Ft.  Leavenworth,  Kan.  Burland  is  chief  of  the 
MCBL  CID  Science  &  Technology  Branch. 


15 


Defense  AT&L:  September-October  2012 


participation.  Additionally,  the  MCBL  makes  every  attempt  to 
synchronize  the  products  being  demonstrated  with  the  CGSC 
course  of  instruction  (e.g.,  handheld  technologies  are  dem¬ 
onstrated  during  brigade  operations).  The  event  is  a  venue 
in  which  developers  from  across  the  Army's  R&D  commu¬ 
nity  receive  feedback  from  experienced  leaders  and  from  the 
many  multinational  and  sister-Service  personnel  attending  the 
CGSC.  Conversely,  the  students  who  participate  in  the  semi¬ 
nars  expand  their  experience  base  and  are  better  prepared 
when  the  new  technologies  arrive  in  the  field. 

The  MCBL,  with  support  from  the  CGSC,  executes  two  to  three 
seminars  each  year,  soliciting  operational  feedback  from  CGSC 
faculty,  students,  and  staff.  The  CGSC  provides  intermediate 
level  education  (ILE)  for  Army  and  sister-Service  officers,  in¬ 
teragency  representatives,  and  international  military  officers. 
The  ILE  course  of  instruction  is  a  10-month,  graduate-level 
program;  the  curriculum  includes  instruction  on  leadership 
philosophy,  military  history,  and  the  military  planning  and  de¬ 
cision-making  processes.  There  are  two  ILE  classes  per  year; 
the  first  begins  in  August  and  ends  in  June,  and  the  second 
begins  in  March  and  ends  in  December. 

With  the  exception  of  two  2-month  periods,  there  are  at 
any  time  approximately  1,400  CGSC  students  (Army,  sister- 
Service,  and  international)  completing  the  CGSC  instruction. 
Given  the  current  operational  environment,  the  average  CGSC 
Army  major  has  deployed  three  times  to  a  combat  zone.  A 
more  experienced  and  expansive  population  does  not  exist 
anywhere  else  in  the  Army.  This  and  the  genuine  need  for 
broad  operational  input  to  S&T  development  work  were  the 
key  drivers  to  the  program. 

The  MCBL  sponsors  and  is  involved  in  a  number  of  S&T  devel¬ 
opment  efforts.  With  the  large  number  of  S&J  projects  across 
the  government,  academia,  and  industry,  it  is  possible  for  the 
MCBL  to  target  and  sponsor  the  MC  technologies  with  the 
greatest  potential  to  fill  the  key  capabilities  gaps  and  tech¬ 


nology  shortfalls  identified  by  evolving  documents,  such  as 
the  Mission  Command  Capabilities  Based  Assessment  (CBA), 
Program  Executive  Officer  Technology  Transition  (T2)  Matrices 
and  other  Joint  Capabilities  Integrated  Development  System 
(JCIDS)  documents,  such  as  the  Net-enabled  MC  Initial  Ca¬ 
pabilities  Document  (Net-enabled  MC  ICD).  All  these  docu¬ 
ments  describe  capability  gaps,  while  the  Net-enabled  MC 
ICD  and  the  Mission  Command  Essential  Capabilities  (MCEC) 
Document  highlight  the  most  critical  ones.  In  an  average  year, 
the  MCBL  participates  in  approximately  seven  S&T  projects. 
In  most,  the  MCBL  provides  comprehensive  operational  input 
and  oversight. 

One  of  the  most  technically  diverse  seminars  was  in  March 
2011.  It  highlighted  technologies  under  development  by  the 
Defense  Advanced  Research  Projects  Agency  (DARPA)  and 
the  Army  Research,  Development  and  Engineering  Command 
(RDECOM).  RDECOM  developed  a  prototype  handheld  appli¬ 
cation  and  demonstrated  just  a  hint  of  the  power  of  augmented 
reality  (AR).  In  this  tool,  the  RDECOM  developers  leveraged 
the  highly  successful  tactical  ground  reporting  (TiGR)  system 
database,  rendering  symbols  and  key  operational  information 
on  a  handheld  (Android)  device.  The  CGSC  officers  could  see 
a  display  of  notional  TiGR  events  (e.g.,  lEDs,  targets,  and  key 
terrain)  close  to  their  location,  overlaid  on  the  screen  and  in¬ 
dicating  general  information  and  geo-location  of  the  events. 

The  application  leveraged  the  internal  capabilities  of  the  hand¬ 
held  device,  including  the  forward-looking  camera  and  the 
global  positioning  system  (GPS).  In  operation,  the  AR  screen 
simply  looked  like  a  window  highlighting  whatever  was  in  the 
field  of  view  of  the  camera  (where  the  device  was  pointing) 
and  those  TiGR  events  within  a  specified  range  of  the  applica¬ 
tion.  Duringthe  demonstration,  CGSC  officers  were  allowed  to 
use  the  handheld  device  and  offer  their  thoughts.  Even  though 
this  research  was  immature,  it  exposed  Army  officers  to  one 
possibility  for  enhanced  and  timely  situational  awareness  at 
the  lowest  tactical  levels. 


"We  work  hard  to  engage  soldiers  on  our 
R&D  efforts  at  every  opportunity,  but  re¬ 
ceiving  input  and  feedback  from  experi¬ 
enced  Army  officers  can  be  challenging 
for  us,"  said  Cyndi  Carpenter,  the  COBRA 
ATO  manager  for  CERDEC's  Command, 
Power  and  Integration  Directorate.  "The 
relationship  the  MCBL  has  built  with  the 
CGSC  and  initiating  the  S&T  Brownbag 
Lunch  program  is  certainly  a  great  use 
of  the  wealth  of  operational  expertise  on 
Fort  Leavenworth.  Our  programs  are,  and 
will  continue  to  be,  better  for  it." 

A  second  technology  demonstrated  at 
the  March  2011  event  focused  on  a  two- 
way,  speech-to-speech  translation  sys¬ 
tem  called  TRANSTAC.  It  had  the  capa- 
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bility  to  translate  Pashto,  Dari,  and  Iraqi  Arabic 
to  English  and  vice  versa.  The  TRANSTAC  de¬ 
vice  was  developed  in  response  to  a  need  from 
Army  units.  Soldiers  at  checkpoints  and  out  on 
patrols  required  the  ability  to  better  communi¬ 
cate  with  the  local  population.  TRANSTAC  of¬ 
fers  a  unique  capability  to  quickly  break  down 
the  language  barrier.  Soldiers  can  speak  into  the 
device  in  English  and  within  a  few  seconds,  both 
oral  and  written  Pashto,  Dari  or  Iraqi  Arabic  is 
relayed  to  the  recipient.  It  can  also  do  the  same 
translation  from  verbal  input  from  any  of  the 
three  languages  and  provide  audible  and  writ¬ 
ten  translation  in  English. 


TRANSTAC  worked  very  well;  at  one  point,  an 
Afghani  CGSC  student  did  an  impromptu  dem¬ 
onstration  of  its  ability  to  translate  Pashto  to 
English.  The  demonstration  highlighted  potential  techniques 
and  word  usage  requirements  in  situations  where  warfighters 
encounter  individuals  using  different  dialects,  as  well  as  the 
need  for  being  prepared  if  the  translation  is  incorrect. 


Post  of  the  Future  (CPOF)  by  extending  CPOF-like  capabili¬ 
ties  to  disadvantaged  users  through  the  maneuver  widget.  It 
will  help  to  ensure  critical  situational  awareness  information 
is  shared  with  all  members  of  the  operational  team,  including 
members  outside  the  Army. 


The  participants  were  particularly  interested  in  the  TRANS¬ 
TAC  product,  and  the  CGSC  officers  were  able  to  see  firsthand 
an  S&T  product  already  being  used  in  theater.  At  the  time  of 
the  demonstration,  DARPA  and  the  Army  had  fielded  102  of 
the  TRANSTAC  devices  to  a  brigade  combat  team  deployed 
to  Afghanistan.  TRANSTAC  received  a  great  deal  of  positive 
feedback  during  the  seminar.  In  fact,  it  subsequently  received 
approval  as  a  System  Under  Evaluation  (SUE)  at  Network  In¬ 
tegration  Event  (NIE)  12.1  and  received  an  evaluation  of  "field 
and  continue  to  develop"  by  the  Brigade  Modernization  Com¬ 
mand  (BMC)  and  Department  of  the  Army.  Brig.  Gen.  Randal 
A.  Dragon,  commander  of  the  BMC,  wrote  of  the  TRANSTAC 
system:  "Machine  Foreign  Language  Translation  (MFLT)  com¬ 
municated  successfully  in  foreign  languages  and  was  easy  to 
install  and  use.”  TRANSTAC  is  scheduled  to  transition  to  the 
Machine  Foreign  Language  Translation  System  (MFLTS)  pro¬ 
gram  of  record.  The  demonstration  helped  highlight  TRANS- 
TAC's  functionality  for  CGSC  officers,  it  also  facilitated  feed¬ 
back  and  identified  areas  where  additional  refinement  could 
enhance  the  tool. 

In  the  most  recent  lunch  event,  the  MCBL  coordinated  with 
the  Program  Manager  Mission  Command  (PM  MC)  to  intro¬ 
duce  its  concept  for  reducing  the  numbers  of  MC  systems 
and  providing  MC  capabilities  to  disadvantaged  users— the 
personnel  on  a  staff  who  are  not  fielded  to  an  MC  system. 
Called  Command  Web,  it  is  a  key  element  of  the  MC  conver¬ 
gence  strategy  for  a  consolidated  web-based  environment. 
Command  Web  is  essentially  a  framework  for  housing  light¬ 
weight  web  applications  called  "widgets"  that  use  the  Ozone 
Widget  Framework  (OWF)  environment.  A  primary  benefit  is 
that  it  allows  third-party  widget  developers  to  build  widgets 
for  a  wide  spectrum  of  users  and  functions— all  interoperable. 
Most  importantly,  this  product  complements  the  Command 


During  the  seminar,  the  PM  MC  developers  received  valuable 
feedback  from  more  than  50  personnel.  Discussions  and  feed¬ 
back  from  experienced  soldiers  centered  on  key  functionality 
within  the  widgets,  interface  designs,  applicability  across  dif¬ 
ferent  computing  environments  (e.g.,  command  post),  and 
general  employment  in  an  operational  environment.  Given 
that  Command  Web  was  still  in  development,  this  feedback 
was  extremely  valuable,  helping  to  ensure  the  fielded  product 
better  meets  the  soldiers'  needs. 

With  the  continual  evolution  of  military  operations  and  an  ever- 
changingtechnical  environment,  the  importance  and  value  of 
bringing  experienced  users  in  early  and  often  in  a  product's 
development  cannot  be  overemphasized.  The  MCBL  recog¬ 
nizes  and  embraces  this  mandate,  reaching  out  to  the  larg¬ 
est  population  of  experience  soldiers  in  the  Army  Command 
and  General  Staff  College.  Through  the  seminar  program,  the 
MCBL  is  able  to  solicit  input  from  CGSC  students  and  staff 
in  a  non-threatening,  volunteer  environment,  where  both  the 
materiel  developer/S&T  organization  and  the  CGSC  student 
can  benefit.  The  three  S&T  Seminars/Brownbag  Lunch  ses¬ 
sions  in  less  than  2  years  have  demonstrated  the  value  of  the 
program,  with  hundreds  of  CGSC  officers  and  staff  participat¬ 
ing  and  six  significant  S&T  programs  gleaning  valuable  input 
for  their  development  efforts.  With  support  from  the  CGSC, 
the  MCBL  will  continue  to  execute  this  unique  program,  which 
serves  as  a  conduit  between  the  S&T  materiel  developers  and 
the  unequaled  experience-base  at  the  CGSC.  The  long-term 
impacts  of  this  effort  are  many— from  improved  products  and 
functionality  to  reduced  life  cycle  costs.  & 

The  authors  can  be  reached  atjeffrey.from@us.army.mil  and  brett. 
burland@us.army.mil. 
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The  Fallacy 
of  Quantifying  Risk 

David  E.  Frick,  Ph.D. 


early  every  article  you  see  in  industry  and  within  the  DoD  literature  on  the  topic  of  "risk 
management"  demonstrates,  advocates,  or  aggrandizes  the  attempt  to  quantify  risk. 
One  might  think  that  if  risk  management  was  truly  a  science  and  uncertainty  could 
be  systematically  quantified  in  some  manner,  then  the  maturity  of  the  profession  of 
project  or  program  management,  as  measured  by  the  number  of  projects  or  programs 
that  meet  cost,  schedule,  and  performance  goals,  would  increase  over  time.  Alas,  the  profession 
is  not  able  to  make  this  claim. 


A  recent  article  in  a  professional  publication  was  no  exception  and  prompted  this  response.  The  author  suggested 
that  a  key  to  risk  analysis  was  "choosing  the  right  technique"  of  quantifying  risk.  The  weakness  in  this  argument 
stems  not  from  the  assertion  that  one  approach  may  be  superior  to  another,  but  rather  from  the  basic  assumption 


Frick  is  a  35-year  veteran  of  the  Department  of  Defense.  He  is  currently  an  acquisition  professional  and  active  risk  practitioner. 
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that  risk  (arising  from  uncertainty)  in  every  situation  can,  in 
fact,  be  quantified. 

One  profession-wide  barrier  to  a  more  meaningful  discussion 
on  this  topic  is  our  collective  looseness  in  the  use  of  language. 
The  lack  of  a  common  taxonomy  serves  to  exacerbate  this  bar¬ 
rier.  In  this  same  article,  the  terms  "risk  analysis"  and  "assess¬ 
ment  of  uncertainty"  appeared  to  represent  the  same  concept. 
They  are  two  distinct  concepts.  The  word  "risk"  is  the  feeble 
attempt  by  humans  to  define  the  ephemeral  abstraction  of 
uncertainty  as  a  tangible  term.  "Risk,"  as  we  commonly  use 
the  term,  is  inherently  unquantifiable. 

Quantitative  techniques  rely  on  empirical  data,  or  at  least 
highly  defensible  estimates.  When  you  discuss  the  uncer¬ 
tainty  of  the  weather  or  solar  flares,  the  historical  data  are 
sufficient  to  make  assertions  that  do  not  cause  statisticians 
to  cringe.  When  you  discuss  the  uncertainty  surrounding  a 
first-time,  never-to-be-repeated  project  or  a  major  systems 
acquisition  (MSA)  designed  to  rely  on  technology  that  has 
not  reached  maturity,  germane  historical  data  seldom  exist. 
One  is  truly  in  a  state  of  not  knowing  what  is  not  known.  Esti¬ 
mates  presented  to  the  Milestone  Decision  Authority  (MDA), 
based  on  experience  and  judgment,  do  have  their  value— but 
the  inexactness  of  most  quantitative  assessments  of  the  true 
state  of  uncertainty  surrounding  programs  makes  predictions 
based  on  these  analyses  no  more  useful  than  flipping  a  coin. 
Estimates  may  be  all  we  have,  but  we  should  not  impute  to 
them  some  characteristic  of  certainty  that  does  not  exist. 

Other  disciplines,  such  as  the  insurance  industry  and  medicine, 
use  the  same  term  to  represent  concepts  dissimilar  to  DoD's 
use  of  the  term.  The  differences  are  subtle  yet  critical,  and 
these  subtle  differences  confound  us. 

In  the  insurance  industry,  years,  decades,  and  centuries 
of  historical  data  give  actuaries  high  confidence  in  making 
generalizations  of  aggregates.  While  no  insurance  company 
can  declare  honestly  that  any  given  man,  born  in  1955,  non- 
smoker  in  good  health,  will  live  another  23.26  years,  it  can 
declare  with  the  utmost  certainty  that  on  average,  all  men 
in  this  category  will.  These  actuaries  bet  their  companies' 
financial  health  on  the  ability  to  accurately  interpret  a  large 
amount  of  historical  data. 

In  medicine— in  the  United  States,  anyway— drug  companies 
spend  billions  of  dollars  annually  to  gather  data.  Test  popula¬ 
tions  only  number  in  the  hundreds  and  sometimes  the  thou¬ 
sands,  but  thanks  to  the  beauty  of  the  law  of  truly  large  num¬ 
bers— with  a  sample  size  large  enough,  any  outrageous  thing  is 
likely  to  happen— we  can  be  confident  that  properly  conducted 
studies  will,  in  fact,  uncover  almost  all  of  the  unintended  con¬ 
sequences  of  a  drug's  effects.  However,  in  DoD  acquisitions, 
we  cannot  be  as  confident  for  a  very  simple  reason— relevant 
historical  data  for  first  time,  never  to  be  repeated  programs 
do  not  exist.  Yes,  we  have  ample  data  on  programs,  in  general, 
but  each  program  is  unique,  will  face  unique  challenges,  and 


will  involve  a  unique  set  of  people.  The  dissimilarities  vastly 
outnumber  the  similarities. 

Pronouncements  that  risk  registers,  quantitative  techniques,  or 
milestone  reviews  "reduce  the  number  of  risks"  demonstrate 
another  fundamental  misunderstanding  of  uncertainty.  First, 
in  the  current  parlance  and  practice,  the  term  risk  "should"  be 
associated  with  numeric  value— a  composite  of  the  probability 
that  a  specific  threat  will  manifest  and  the  impact  of  that  mani¬ 
festation.  In  program  management,  we  are  concerned  with  the 
impact  on  cost,  schedule,  or  performance.  We  all  recognize 
the  equation  Risk  =  Probability  x  Impact— or  some  pair  of  the 
terms  potential,  likelihood,  damage,  effect,  and  consequences. 
Probability  is  a  number  (0.0  to  1.0).  Impact  is  usually  visualized 
as  something  that  can  be  measured,  e.g.,  dollars  (cost);  hours, 
days,  or  weeks  (schedule);  or  customer  satisfaction,  quality, 
speed,  durability,  mean  time  between  failures  (performance). 
Therefore,  "risk"  should  be  defined  in  terms  of  one  or  more 
specific  units. 

For  example,  the  result  of  some  event  might  have  an  impact 
of  $10,000  plus  4  days  schedule  slip  plus  a  10  percent  reduc¬ 
tion  in  system  performance.  Instead,  in  DoD,  we  choose  to 
place  probability  in  one  of  never  more  than  five,  overly  simplis¬ 
tic  buckets— very  low  (1),  low  (2),  medium  (3),  high  (4),  and 
very  high  (5);  then  we  do  the  same  to  impact.  The  product  of 
these  assignments  is  a  number  in  the  range  of  1  to  25.  Then, 
we  arbitrarily  slice  this  range  into  three  sections  and  name 
them  low,  medium,  or  high.  Talk  about  excessive  aggregation! 
Can  you  imagine  an  insurance  company  only  offering  three 
premium  levels  to  a  population  as  diverse  as  ours?  Such  an 
approach  would  not  endanger  the  insurance  company,  if  its 
client  base  was  large  enough,  but  I  suspect  that  discriminat¬ 
ing  consumers,  at  least  those  in  the  low  risk  categories,  would 
shop  elsewhere. 

Second,  while  "risk  mitigation"  may  reduce  the  total  number  of 
threats  (by  reducing  probability  or  impact  of  a  specific  threat 
to  zero),  what  the  practitioner  usually  means  is  that  the  value 
of  the  risk  for  a  specific  post-mitigation  threat  is  so  inconse¬ 
quential  that  it  no  longer  merits  an  expenditure  of  brainpower. 
Nonetheless,  the  specific  threat  still  exists  and  even  the  highly 
improbable  event  does  occasionally  manifest.  Nassim  Taleb 
refers  to  this  as  the  "black  swan"  event. 

Third,  while  eliminating  a  single  threat  from  consideration  may 
have  value,  if  you  consider  the  near  infinite  number  of  threats 
that  may  affect  a  program  but  are  not  being  considered  be¬ 
cause  they  are  so  remote  in  possibility  or  simply  not  known  or 
knowable,  suggesting  that  a  specific  program  faces  no  more 
than  10, 100,  or  even  1,000  "risks"  is  naivete. 

In  the  "identification  phase"  of  "risk  analysis"  (better  named 
threat  identification),  practitioners  are  wont  to  stop  identifying 
threats  at  some  arbitrary  point,  usually  the  number  of  lines  that 
fit  on  the  risk  slide  in  some  PowerPoint  presentation.  Admit¬ 
tedly,  there  is  a  point  at  which  the  cost  of  committing  threats 
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The  inexactness  of  most 
quantitative  assessments  of 
the  true  state  of  uncertainty 
surrounding  programs  makes 
predictions  based  on  these 
analyses  no  more  useful  than 
flipping  a  coin. 


to  paper  exceeds  prudence.  A  human  extinction-level  event 
(such  as  a  massive  meteor  strike)  would  likely  have  devastat¬ 
ing  consequences  on  your  project,  program,  or  MDA.  This 
threat  always  exists,  but  expending  time  and  effort  thinking 
about  it  (or  reporting  it  to  the  MDA)  would  probably  not  be 
prudent.  The  question  is  how  many  low-probability/high-im- 
pact  threats  are  not  being  considered  simply  because  some 
risk  analyst  ran  out  of  lines  in  the  risk  register  or  simply  failed 
to  identify  them? 

Furthermore,  while  uncertainty  comprises  the  totality  of  pos¬ 
sible  good  things  (opportunities)  and  bad  things  (threats), 
invariably,  most  risk  management  practitioners  only  consider 
the  bad  things.  I  laud  DoD  and  the  Project  Management  Insti¬ 
tute  (PMI)  for  stressing  this  point  by  stating  in  the  Risk  Man¬ 
agement  Guide  to  DoD  Acquisition,  6th  Edition,  and  the  PMBOK 
Guide— Fourth  Edition,  that  the  objectives  of  risk  management 
are  to  increase  the  probability  and  impact  of  positive  events 
and  decrease  the  probability  and  impact  of  negative  impacts. 
Nonetheless,  in  the  common  parlance,  risk  continues  to  be 
synonymous  with  the  consequences  of  the  negative.  For  myr¬ 
iad  reasons,  the  discussion  of  potential  opportunities  tends  to 
get  short  shrift. 

The  issue  becomes  more  absurd  in  risk  averse  organizations. 
There  is  nothing  objectionable  to  an  organization  being  risk 
averse,  especially  in  response  to  the  contemporaneous  pro¬ 
pensity  of  Congress,  but  when  the  analyst  allows  a  conserva¬ 
tive  trend  to  influence  the  analysis  of  a  project's  or  program's 
potential  success,  the  program  management  profession  is 
harmed.  Big  risk-big  reward  may  be  a  good  cliche  for  the  mis¬ 
sion  statement,  but  the  culture  of  the  organization  will  more 
strongly  influence  the  final  risk  assessment  than  the  printed 
strategic  plan.  High-impact  threats  are  often  hidden  or  ig¬ 
nored.  Estimates  are  viewed  through  the  lens  of  the  best  case 
scenario.  The  MDA  then  makes  decisions  based  on  informa¬ 
tion  that  is  incomplete,  so  more  programs  fail  than  anticipated. 


Risk  handling  and  risk  mitigation,  also 
terms  without  precise  universally-ac¬ 
cepted  definitions,  are  terms  commonly 
thrown  about  by  program  management 
practitioners  to  justify  removing  a  spe¬ 
cific  identified  threat  from  the  few  listed 
in  the  risk  register.  Both  PMI  and  DoD 
identify  four  risk  mitigation  techniques: 
avoiding  (eliminating  the  threat  or  con¬ 
sequence),  reducing  (the  probability  or 
consequences  of  the  threat  manifest¬ 
ing),  transferring  (this  method  is  a  bit 
nebulous,  but  view  it  as  making  the  threat 
someone  else's  problem,  e.g.,  insurance), 
or  assuming  (the  risk). 

Consider,  instead,  the  proposition  that 
from  the  perspective  of  the  major  pro¬ 
gram,  there  exist  only  two  categories  of 
action  to  handle  or  manage  risk: 

■  Reduce  the  composite  risk  index.  This  means  taking  some 
action  within  the  limits  of  available  knowledge  and  resources 
that  decreases  the  probability  of  a  threat  manifesting  (hope¬ 
fully  to  zero)  or  reducing  its  impact  (again,  hopefully  to  zero). 

■  Assuming  the  risk,  when  probability  and  impact  are  both 
greater  than  zero. 

All  actions  under  the  rubric  of  "risk  mitigation''  or  risk  handling 
fall  in  this  first  category.  Risk  avoidance,  e.g.,  deciding  not  to 
start  a  program,  is  one  manner  of  reducing  the  probability  of 
the  threat  to  zero.  Risk  transfer,  e.g.,  insurance,  reduces  the 
impact  from  the  perspective  of  the  program  to  near  zero.  To 
stress  the  point,  risk  mitigation  "always”  has  a  cost,  e.g.,  ex¬ 
penditure  of  resources  or  the  ephemeral  opportunity  costs. 
Risk  mitigation  becomes  a  recursive  exercise  in  cost-benefit 
analyses.  In  the  end,  when  all  efforts  at  mitigating  risk  have 
been  exhausted  or  evaluated  as  too  costly  for  the  potential 
benefit  and  the  probability  and  impact  of  a  specific  threat 
is  still  greater  than  zero,  the  only  recourse  left  is  the  second 
category— to  assume  the  risk.  Assuming  risk  should  not  be 
considered  bad  leadership.  On  the  contrary,  history  is  replete 
with  examples  of  commanders  assuming  great  risk  (usually 
arising  from  lack  of  information  about  the  enemy),  yet  achiev¬ 
ing  great  outcomes. 

Attempts  at  quantifying  risk  are  not,  in  and  of  themselves, 
objectionable.  Prudence  demands  that  program  manage¬ 
ment  practitioners  quantify,  to  the  greatest  extent  practica¬ 
ble,  and  prioritize  known  threats  so  that  limited  resources  can 
be  applied  in  a  thoughtful  manner  to  reduce  the  component 
probabilities  and  impacts.  On  the  other  hand,  the  practice 
of  stating  to  some  level  of  surety  that,  based  on  some  eso¬ 
teric  risk  analysis,  program  risk  is  low,  medium,  or  high,  is 
damaging  to  the  program  management  profession.  The  un¬ 
expected,  harmful  "black  swan”  event  can  suggest  to  those 
not  well  schooled  in  risk  management/risk  analysis  that  the 
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offered  analysis  was  incomplete,  incompetent, 
faulty,  or  dishonest— not  good  for  the  program 
management  profession  and  replete  with  con¬ 
sequences,  e.g.,  Nunn-McCurdy  reviews. 

Practitioners  would  be  much  better  served 
to  be  more  complete  in  acknowledging  and  re¬ 
porting  the  complete  state  of  uncertainty  in  a 
project.  The  output  of  a  complete  risk  analysis 
should  include: 


The  number  of  threats  identified  in  threat 
identification 
The  number  of  identified  threats  for  which 
either  probability  or  impact  can  or  have  been 
reduced  to  zero 

The  number  of  identified  threats  for  which 
the  composite  risk  cannot  be  reduced  to  zero 
within  current  resource  constraints  and  must 
be  "assumed” 

An  enumeration  of  the  identified  threats  for  which  the  or¬ 
ganization  has  no  historical  experience 
The  magnitude  of  the  unknown-unknowns.  Of  course,  this 
number  cannot  be  quantified,  but  an  honest,  subjective  as¬ 
sessment  is  much  more  valuable  to  the  MDA  than  is  silence. 
An  assessment  of  project  success.  Again,  this  is  a  highly 
subjective  assertion.  Be  honest.  An  honest,  subjective  as¬ 
sessment  is  much  more  useful  to  the  MDA  than  the  typical, 
overly  optimistic,  agenda-driven  pronouncements. 


Big  risk-big  reward  may  be 
a  good  cliche  for  the  mission 
statement,  but  the  culture 
of  the  organization  will  more 
strongly  influence  the 
final  risk  assessment  than 
the  printed  strategic  plan. 


Human  nature  is  replete  with  cognitive  biases.  Multiple  stud¬ 
ies  have  shown  how  estimates  are  subject  to  the  confound¬ 
ing  influence  of  expectation  bias.  A  can-do  attitude  is  a  great 
characteristic,  unless  it  blinds  the  program  manager  to  the 
obvious  truth.  Take  a  step  back  and  have  the  courage  to  admit 
you  don't  know  what  you  don't  know.  & 
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The  Department  of  Defense  began  using  integrated  product 
teams  (IPTs)  in  weapons  development  in  the  1980s.  Their  use 
proliferated  after  the  secretary  of  Defense  directed  their  use 
under  the  integrated  product  and  process  development  (IPPD) 
concept  in  1995.  Over  the  years,  the  term  IPT  has  been  applied 
to  a  variety  of  groups,  councils,  tiger  teams,  and  boards.  Although  these 
groups  all  have  important  functions,  the  IPT  is  a  specific  and  power¬ 
ful  tool  that  begs  definition.  Properly  chartered,  an  IPT  is  a  protection 
against  obstacles  to  success  I've  observed  in  acquisition  program  offices: 
parochialism,  functional  bias  and  poor  communication,  to  name  a  few. 
These  obstacles  existed  in  spite  of  the  fact  that  the  people  involved  were 
dedicated,  experienced,  and  patriotic. 

I  was  assigned  to  a  major  defense  acquisition  aircraft  program  that  had  operated 
under  strong,  capable,  experienced  functional  alignment.  The  program  office  floor 
space  had  been  physically  arranged  into  functional  areas:  program  management, 
engineering,  contracting,  logistics,  testing,  financial  management,  training  systems, 
manufacturing,  security,  and  administration,  each  headed  by  a  GS-15  or  colonel.  There 
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The  charter  had  to  be 
specific,  not  at  high 
level,  not  vague  or 
timid.  It  had  to  contain 
milestones,  outcomes. 


were  SES-level  functional  heads  at  the  base  level  to  which 
these  functional  chiefs,  and  those  in  other  program  offices  on 
base,  reported.  There  were  also  user  representatives  from  the 
combat  commands  on  base. 

When  a  major  issue,  such  as  a  proposed  design  change,  arose 
in  the  program  office,  each  functional  chief  would  assemble 
his  functional  team  and  formulate  their  best  position  on  the 
issue.  Each  functional  position  then  would  be  presented  in  a 
staff  meeting  to  the  program  manager,  who  would  eventu¬ 
ally  make  the  decision  on  how  to  proceed.  Next,  the  deputy 
program  manager  and  chief  of  contracting  would  travel  to  the 
prime  contractor  to  present  the  decision.  One  of  the  user  reps 
would  check  back  with  their  combat  command  headquarters. 
The  program  manager  ran  the  program  by  being  the  arbiter 
among  valid  but  competing  positions  among  his  functional 
chiefs,  sorting  out  issues  at  his  level. 

I  was  assigned  to  the  program  office  about  the  same  time  that 
a  new  program  manager,  a  general  officer,  was  assigned.  The 
general  quickly  became  overwhelmed  with  having  to  deter¬ 
mine  the  best  direction  for  the  aircraft  program  while  being 
faced  with  conflicting  recommendations  from  his  functional 
chiefs,  often  in  areas  where  he  had  little  experience  himself. 
He  sensed  that  program  decisions  had  been  made  in  the  past 
based  on  the  strength  of  arguments  and  personalities.  He 
believed  this  was  not  always  the  best  balanced  approach  for 
the  airplane  program  overall,  nor  the  most  efficient  applica¬ 
tion  of  the  program  office  expertise.  The  program  manager 
had  just  come  from  a  base  that  had  undertaken  a  base-wide 
transformation  to  IPTs  in  their  program  offices.  This  had  in¬ 
volved  extensive  training,  but  it  had  paid  off  in  efficiency  and 
effectiveness,  and  he  determined  it  was  time  it  install  IPTs  in 
his  program  office. 

At  first  there  was  some  unease  with  the  program  manager's 
IPT  initiative,  but  that  diminished  as  the  functional  chiefs 


each  volunteered  to  lead  an  IPT.  That  is  where  things  got 
interesting.  The  program  manager  was  firm  that  IPTs  would 
be  chartered  by  him  to  manage  or  produce  a  product,  such  as 
a  test  plan  or  engine,  or  a  major  subsystem.  Rather  than  be  an 
engineering  team  or  contracting  team,  the  I  PTs  were  to  repre¬ 
sent  a  product,  not  a  function.  There  would  be  no  engineering 
solutions  or  logistics  positions  or  testing  imperatives.  There 
would  only  be  a  team  solution  for  the  product,  balancing  all 
functional  inputs  at  the  working  level.  The  IPT  organization 
would  replace  the  functional  organization  process  of  handing 
off  a  product  from  one  stovepipe  to  another— from  engineer¬ 
ing  to  manufacturing  to  logistics,  and  back  again. 

He  expected  his  functional  chiefs  to  take  on  a  new  respon¬ 
sibility.  They  were  to  help  him  identify  key  products  or  areas 
in  the  weapon  system  that  needed  an  IPT.  The  manning  de¬ 
mand  for  IPTs  required  they  be  few  as  possible  in  number 
and  cover  major  products.  They  didn't  need  a  seat  cushion 
IPT.  They  were  to  then  help  write  a  charter  for  each  IPT.  Then 
identify  members  from  each  functional  discipline  needed  on 
the  IPT.  Next,  the  functional  heads  were  to  empower  the 
members  they  put  on  an  IPT.  No  running  back  to  the  chief 
engineer  for  mother-may-1.  And  there  was  no  space  for  ob¬ 
servers,  only  necessary  contributors.  (I  am  reminded  of  a 
senior  acquisition  official  who  said  she  only  wanted  members 
on  herteam  who  would  lose  their  jobs  if  the  team  failed.  The 
message  was  no  hangers-on,  no  observers,  and  no  kibitzers.) 
The  IPT  concept  was  decision  making  and  execution  at  the 
lowest  level. 

The  program  manager  expected  some  resistance.  He  remem¬ 
bered  at  his  previous  base  the  IPT  concept  had  required  buy-in 
from  the  senior  functional  heads  on  base,  and  the  senior  of¬ 
ficers  at  the  base.  So  he  met  with  each  senior  stakeholder  on 
base.  He  explained  to  the  senior  leaders  something  that  he 
had  discovered.  The  program  was  in  trouble,  but  no  one  was 
accountable.  The  best  example  of  the  trouble  was  the  test 
schedule  for  the  electronic  countermeasures  (ECM)  system 
was  not  being  met. 

The  logistics  functional  group  would  not  agree  to  allow  test¬ 
ing  to  proceed  until  the  important  maintainability  features 
were  included  early  in  the  test  schedule.  The  engineering 
group  stated  that  testing  must  be  held  up  until  certain  engi¬ 
neering  questions  were  worked  out.  The  testing  organization 
was  not  willing  to  proceed  until  all  testing  criteria  met  their 
developmental  testing  objectives.  The  program  manager 
pointed  out  that  in  this  example— and  there  were  many  oth¬ 
ers— there  was  validity  to  each  position.  But  there  was  no  one 
accountable  for  the  product,  in  this  case,  the  ECM  system. 

He  viewed  the  responsibility  of  each  senior  head  to  be  to  get 
the  airplane  to  the  warfighter.  To  do  that  he  would  use  IPTs 
and  he  solicited  their  help.  The  senior  functional  and  base 
leaders  agreed  to  support  him,  but  not  without  reservations 
and  concerns,  and  doubts.  After  some  discussion,  they  agreed 
to  exercise  their  functional  responsibilities  by  seeing  that  the 
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right  functional  experts  were  assigned  to  the  right  IPTs.  They 
agreed  to  offer  sound  balanced  processes  to  their  IPT  mem¬ 
bers,  but  let  them  manage  the  products. 

Here  is  what  the  program  manager  believed  I  PTs  were  and  what 
they  were  not.  The  letters  in  "IPT”  have  defined  meanings. 

Integrated  means  the  team  is  composed  of  every  specialty  or 
discipline  needed  to  deliver  the  product.  I  was  appointed  lead 
for  the  ECM  IPT.  This  IPT  also  needed  domain  representatives 
in  development,  test,  manufacturing,  contracting,  budgeting, 
integration,  deployment,  and  sustainment.  The  team  included 
the  user  and  the  contractor  during  all  meetings  and  delibera¬ 
tions,  not  as  an  afterthought. 

Product  means  the  team  is  responsible  for  a  product.  It  is  not 
a  review  group  to  monitor  progress  or  a  tiger  team  to  address 
a  single  problem.  The  product  may  be  a  piece  of  equipment  or 
a  test  plan,  but  a  product  must  be  defined.  In  my  case  it  was 
the  ECM  system  that  would  be  provided  as  government  fur¬ 
nished  equipment  (GFE)  to  the  prime  contractor  to  integrate 
into  the  airplane. 

Team  means  that  the  members  work  for  consensus.  A  team 
has  one  leader.  I  was  a  leader  among  peers,  regardless  of  rank 
or  function.  Each  member  had  equal  say.  As  leader  I  did  not 
have  a  technical  or  functional  responsibility.  My  job  was  to 
see  that  the  team  delivered  a  product  that  balanced  factors 
from  all  members,  to  see  the  team  reach  consensus.  To  op¬ 
erate  best  the  team  members  are  collocated,  with  their  own 
meeting  area. 

The  first  step  was  to  determine  the  IPTs.  The  program  man¬ 
ager  and  his  functional  chiefs  decided  which  major  products  or 
components  needed  direct  management  by  an  IPT.  Next  they 
took  the  necessary  time  to  carefully  craft  a  charter  for  each 
IPT.  The  charter  had  to  be  specific,  not  at  high  level,  not  vague 
or  timid.  It  had  to  contain  milestones,  outcomes,  or  specific 
objectives.  The  charter  had  to  state  the  IPT's  authority  and  the 
next  level  of  reporting  for  the  IPT.  The  program  manager  and 
his  chiefs  named  in  the  charter  an  IPT  lead  whose  responsibili¬ 
ties  were  stated,  which  did  not  include  any  functional  responsi¬ 
bilities.  Finally  the  charter  was  signed  by  the  program  manager. 
Each  charter  was  eventually  posted  in  the  IPT's  team  area. 

Next  came  the  naming  of  IPT  members.  Each  must  be  re¬ 
lieved  of  other  duties  sufficiently  to  accomplish  the  objectives 
in  the  charter.  The  chiefs  had  to  assure  the  approval  of  the 
individual's  supervisory  chain.  Finally,  the  IPT  members  must 
be  empowered  to  do  what  is  in  the  charter. 

There  are  a  few  tips  I  learned  as  an  IPT  lead. 

The  IPT  leader  must: 

■  Be  respected  in  and  out  of  the  IPT 

■  Be  balanced 


■  Possess  managerial  skills 

■  Be  able  to  manage  the  external  environment  to  allow  the 
IPT  to  focus  on  their  work 

■  Be  decisive.  Make  the  decision  with  the  best  consensus 
when  the  decision  must  be  made 

■  Not  be  biased  toward  any  functional  or  technical  viewpoint 

The  IPT  members  must: 

■  Have  domain  or  functional  expertise 

■  Be  empowered  and  have  authority  for  their  domain 

■  Be  committed  to  the  IPT's  product  and  charter 

■  Agree  on  ground  rules,  time  demands  and  schedules 

■  Be  open  minded 

■  Be  a  team  player 

Not  every  program  office  will  be  able  arrange  all  the  particu¬ 
lars  I  illustrated  above,  but  the  core  functions  are  achievable. 
You  may  not  have  the  luxury  of  dedicated  meeting  rooms, 
but  you  can  schedule  common  meeting  spaces.  You  may 
not  have  all  members  collocated,  but  there  are  ways  to  still 
meet  together  using  travel,  video  teleconferencing,  or,  as  a 
last  resort,  speaker  phones.  The  essential  requirement  is  that 
all  IPT  members  be  present  at  meetings.  You  may  not  be  able 
to  have  (or  even  need)  full-time  access  to  every  functional 
expert  called  for,  but  you  must  push  for  dedicated  identified 
members,  even  if  part  time.  Two  mandatory  members  of  your 
IPT— and  this  is  essential— are  the  user  and  the  contractor. 
If  your  IPT  is  for  a  GFE  component,  you  need  both  the  GFE 
vendor  and  the  prime  contractor. 

There  were  other  tasks  the  general  faced  to  implement  IPTs. 
The  facility  manager  had  to  rearrange  the  cubicles  so  the  IPT 
members  could  sit  together,  and  so  that  each  IPT  had  a  meet¬ 
ing  area.  The  head  of  human  resources  had  to  agree  to  permit 
each  IPT  lead  to  make  written  input  to  the  appraisals  and  per¬ 
formance  reports  of  his  IPT  members,  such  as  by  formal  letter 
to  the  member's  supervisor  of  record. 

The  IPT  was  tasked,  recognized,  and  rewarded  as  a  team,  not 
as  individuals. 

At  first  there  was  some  uneasiness  and  mistrust  among  the 
IPT  members.  But  as  they  began  to  meet  and  solve  problems 
together  I  witnessed  an  interesting  phenomenon.  They  began 
to  achieve  successes.  Small  organizational  successes  at  first, 
but  then  they  began  to  tackle  and  solve  bigger  challenges. 
They  began  to  learn  each  other's  jobs.  They  became  able  to 
answer  outside  inquiries  for  each  other  when  a  member  was 
not  available.  They  began  to  cover  for  each  other. 

IPTs  do  not  arise  automatically,  or  naturally,  or  spontane¬ 
ously  out  of  need.  Nor  are  they  learned  on  the  fly.  They  must 
be  worked  at  to  work.  There  are  a  variety  of  people  who  can 
and  will  say  no  to  an  idea.  IPT  members  are  empowered  to 
say  yes.  & 

The  author  can  be  reached  at  david.hofstadter@dau.mil. 
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Krieger  is  an  intermittent  professor  of  Contract  Management  in  the  Defense  Systems  Management  College  and  an  independent 
acquisition  consultant.  He  had  31  years  of  government  experience  in  contracting  and  acquisition  before  retiring  from  the  Civil  Service 
in  2010  and  is  a  former  assistant  commander  for  contracts  at  the  Marine  Corps  Systems  Command. 


Faced  with  declining  budgets  and  the  spectre  of  sequestration  under  the  Budget  Control 
Act,  the  Department  of  Defense  must  do  much  more  than  it  has  ever  done  in  the  past 
to  get  more  bang  for  the  buck.  This  is  usually  translated  to  mean  greater  quantities  for 
the  same  funding.  How  do  we  do  that? 

In  1729  Jonathan  Swift  wrote  a  satirical  essay,  A  Modest  Proposal  for  Preventing  the  Children  of  Poor  People  From 
Being  a  Burden  on  Their  Parents  or  Country ,  and  for  Making  Them  Beneficial  to  the  Publick,  which  has  come  to 
be  called  A  Modest  Proposal.  In  Swift's  essay,  the  "modest  proposal"  was  for  the  Irish  to  raise  themselves  up 
from  poverty  by  selling  their  children  as  food  "...  to  the  persons  of  quality  and  fortune  through  the  kingdom." 
As  that  title  was  already  taken,  we  have  called  this  piece  An  Immodest  Proposal  which  next  to  Swift's  this  is 
very  much  so.  The  other  choice  would  have  been  A  Truly  Modest  Proposal,  which  this  certainly  is. 

If  Only  DoD  Operated  as  a  Business 

In  Breaking  the  Camel's  Back  ( Defense  AT&L:  July-August  2009),  Roy  Wood  and  I  took  exception  to  a  state¬ 
ment  in  a  2008  report  from  the  Defense  Science  Board,  "DoD's  business  practices  need  not  be  worse  than  the 
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commercial  sector's  norm.”  It  was  that  statement  that 
broke  the  camel's  back,  as  far  as  we  were  concerned. 
We  wrote: 

If  the  Department  of  Defense  could  operate  more  like 
a  commercial  business,  weapons  systems  would  be 
cheaper,  on  time,  and  meet  the  needs  of  the  battlefield 
commanders.  That  is  a  recurring  theme  in  the  dozens 
of  acquisition  reform  studies  over  the  past  several  de¬ 
cades.  Most  recently,  a  Defense  Science  Board  report 
noted  almost  wistfully  that  DoD  should  adopt  "com¬ 
monplace  tenets  of  good  management  practice  that 
abound  in  the  commercial  sector"  (Defense  Science 
Board  2008  report,  "Defense  Imperatives  for  the  New 
Administration").  While  no  one  we  know  has  the  chutz¬ 
pah  to  defend  many  of  DoD's  more  notorious  business 
blunders,  comparing  DoD  with  commercial  business  is 
a  faulty  analogy.  While  "making  DoD  work  more  like  a 
business"  makes  for  a  good  sound  bite,  it  grossly  over¬ 
simplifies  the  situation  and  can  inadvertently  drive  dis¬ 
cussion  away  from  realistic  solutions.  To  apply  a  quote 
from  H.L.  Mencken:  "There  is  always  a  well  known  so¬ 
lution  to  every  human  problem— neat,  plausible,  and 
wrong." 

We  went  on  to  argue  why  the  Department  of  Defense 
is  not  like  a  business,  and  should  not  be  expected  to 
operate  as  one.  However,  the  country  as  a  whole,  and 
the  Department  in  particular,  are  facing  dire  financial 
straits.  We  must  now  explore  every  nook  and  cranny  to 
find  ways  to  help  ends  meet.  Where  Wood  and  I  had  ar¬ 
gued  that  the  Department  had  no  profit  motive  to  drive 
its  behavior,  we  must  now  create  that  profit  motive,  or 
something  that  can  stand  in  its  stead. 

Making  a  ‘Profit’  from  Department  of 
Defense  Programs 

Let's  begin  with  a  discussion  of  profit.  As  we're  in  the 
business  of  acquisition,  let's  look  to  the  Federal  Acqui¬ 
sition  Regulation  (FAR).  Interestingly  enough,  the  FAR 
does  not  define  "profit,”  so  we  must  look  elsewhere. 
According  to  The  Government  Contracts  Reference  Book , 
profit  is  "The  amount  realized  by  a  contractor  after  the 
costs  of  performance  (both  direct  and  indirect)  are  de¬ 
ducted  from  the  amount  to  be  paid  under  the  terms  of 
the  contract.”  For  the  Department  of  Defense,  it  would 
be  something  like,  "The  amount  realized  by  the  Govern¬ 
ment  after  the  costs  of  performance  (both  direct  and 
indirect)  are  paid.”  Now  in  the  best  of  all  worlds,  that 
would  be  something  to  achieve.  At  a  lesser  level,  if  the 
Department  could  just  defray  even  a  small  percentage 
of  program  costs  through  recoupment  of  expenditures, 
we  would  be  much  better  off. 


Now,  "recoupment”  is  a  word  defined  in  the  FAR.  Ac¬ 
cording  to  FAR  35.001  Definitions,  "Recoupment,”  as 
used  in  this  part,  means  the  recovery  by  the  Govern¬ 
ment  of  Government-funded  nonrecurring  costs  from 
contractors  that  sell,  lease,  or  license  the  resulting 
products  or  technology  to  buyers  other  than  the  Fed¬ 
eral  Government. 

And  that,  ladies  and  gentlemen,  is  what  this  article  is 
about,  only  more  so.  I  would  argue  that  where  we  can, 
we  should  not  only  recoup  the  nonrecurring  costs,  but 
also  the  recurring  costs.  In  fact,  we  should  go  one  step 
further,  to  the  degree  that  selling,  leasing,  or  licensing 
the  resulting  products  or  technology  to  buyers  other 
than  the  Federal  Government  recoups  all  Govern¬ 
ment-funded  nonrecurring  costs  and  recurring  costs, 
and  there  are  additional  sales,  leases,  or  licenses,  the 
Government  should  take  a  percentage  of  those  and  ei¬ 
ther  return  the  money  to  the  Treasury  or  use  to  defray 
the  cost  of  other  Government  programs.  Such  bold  an 
approach  would  require  legislation,  as  the  law  and  32 
CFR  §  165.6  require,  "Recovery  of  nonrecurring  cost 
recoupment  charges  shall  cease  upon  the  recovery  of 
total  DoD  costs.” 

Getting  Our  Money’s  Worth 

At  this  point  you  are  probably  skeptical  about  the  pos¬ 
sibility  of  such  a  fantastic  construct  being  viable.  Let's 
explore  one  example  and  see,  the  Global  Positioning 
System  (GPS).  (Admittedly,  it  may  be  one  of  the  easi¬ 
est  examples.)  At  this  time,  the  Department,  through  the 
Global  Positioning  Systems  Directorate,  acquisition  of¬ 
fice  for  developing  and  producing  GPS  satellites,  ground 
systems  and  military  user  equipment,  is  pursuing  the 
future  of  GPS.  This  is  being  done  through  contracts  with 
the  Boeing  and  Lockheed  Martin.  Boeing  has  contracts 
for  development  and  production  of  33  GPS  1 1 F  space  ve¬ 
hicles  (SVs).  The  first  GPS  1 1 F  was  launched  on  May  27, 
2010.  In  May  2008,  Lockheed  Martin  was  awarded  the 
first  GPS  III  increment  contract,  for  the  development  and 
production  of  two  initial  SVs,  with  options  for  up  to  ten 
additional  SVs.  All  of  that  has  got  to  cost  a  lot  of  money. 
How  much?  Well,  according  to  Coleman  Bazelon  of  The 
Brattle  Group,  Inc.,  "The  Federal  government  has  spent 
an  estimated  $35  billion  on  the  current  GPS  network 
since  it  began  GPS  operations  in  the  mid-1970s.”  How  do 
we  get  that  back?  Or  pay  for  the  next  GPS  constellation? 

Let's  look  at  what  we  might  be  able  to  achieve  in  the 
future,  if  we  were  to  go  about  things  smartly.  GPS  is  big 
business,  really  big.  Although  the  first  thing  that  comes 
to  mind  about  GPS  is  navigation,  GPS  is  ubiquitous, 
even  reaching  into  areas  such  as  banking  and  invest- 


merits,  through  computer  clock  synchronization.  Wikipedia 
lists  a  myriad  of  civilian  applications  for  GPS  (i.e.,  clock  syn¬ 
chronization,  cellular  telephony,  disaster  relief/emergency 
services,  geofencing,  geotagging,  GPS  aircraft  tracking, 
GPS  tours,  map-making,  navigation,  phasor  measurements, 
robotics,  recreation,  surveying,  tectonics,  telematics,  fleet 
tracking).  That's  a  lot  of  applications,  and  there  are  billions 
of  dollars  associated  with  those  applications.  Just  think  of 
the  number  of  users  for  a  single  application.  According  to  a 
June  2010  column  in  Information  Week ,  "The  number  of  traffic 
information  users  globally  is  expected  to  grow  to  more  than 


Twenty  years  ago,  AM  General  was  able  to  commercialize 
the  M998  High  Mobility  Multipurpose  Wheeled  Vehicle 
(HMMWV),  the  "Humvee."  Later  owned  by  General  Motors, 
the  line  included  luxury  4x4  vehicles,  sport-utility  vehicles 
(SUVs)  and  sport-utility  trucks  (SUTs).  Now,  think  about  all 
the  vehicle  developments  that  have  been  going  on,  or  are  on¬ 
going,  in  the  Army  and  the  Marine  Corps  as  a  result  of  years 
of  war  and  the  ongoing  reset. 

Or  think  about  the  ongoing  development  of  unmanned  aerial 
vehicles  (UAVs),  both  fixed-wing  and  rotary-wing,  which  have 


Suppose  that  each  of  those  370  million  traffic 
information  users  had  just  $10.00  added  to  the  purchase 

t 

price  of  their  device...a  relative  drop  in  the  bucket,  but 
multiplied  by  370  million  is  equal  to  $3.7  billion. 


370  million  by  2015,  up  from  57  million  this  year,  according 
to  new  data  from  ABI  Research,  which  also  found  that  traffic 
information  remains  the  most  important  feature  of  mobile 
navigation  services." 

Now,  do  some  simple  math.  Just  suppose  that  each  of  those 
370  million  traffic  information  users  had  just  $10.00  added 
to  the  purchase  price  of  their  device,  transparent  to  the  pur¬ 
chaser.  A  relative  drop  in  the  bucket,  but  multiplied  by  370 
million  is  equal  to  $3.7  billion.  Or,  a  single  dollar  added  to 
a  monthly  phone  bill  for  each  of  the  327,577,529  wireless 
subscriber  connections  that  CTIA-The  Wireless  Associa¬ 
tion  says  there  are  in  the  United  States.  (Note:  That  num¬ 
ber  exceeds  the  population  of  the  United  States.)  Multiply 
$327,577,529  a  month  by  12  months,  and  you  get  $3.9  billion 
a  year.  Now  add  the  two  numbers  we've  calculated,  think 
about  potential  recoupment  from  other  application  errors, 
and,  voila,  we  have  a  new  GPS  constellation  in  no  time.  As 
Sen.  Everett  Dirksen  is  reputed  to  have  said,  "A  billion  here, 
a  billion  there,  and  pretty  soon  you're  talking  real  money." 
This  sort  of  recoupment  could  easily  be  administered  in  the 
same  way  as  the  excise  tax  on  gasoline  and  diesel  fuel  or 
sales  taxes. 

Low-Hanging  Fruit  and  Higher-Hanging  Fruit 

Admittedly,  GPS  represents  a  target  that  is  easily  achievable, 
and  which  would  not  require  a  lot  of  effort  (i.e.,  low-hanging 
fruit).  However,  there  are  many  trees  in  an  orchard,  and  even  if 
some  of  the  fruit  might  be  somewhat  more  difficult  to  harvest, 
it  could,  ultimately,  be  worth  the  effort. 


come  a  long,  long  way  since  the  original  Kettering  Bug  in  1918. 
With  oversight  from  the  Federal  Aviation  Administration, 
which  will  eventually  come,  UAVs  can  become  workhorses 
for  commercial  enterprises  ranging  from  cattle  ranching  to 
firefighting.  Just  imagine  a  rancher  launching  a  hand-launched 
UAV  to  explore  his  fence  line  for  breaks,  or  a  firefighter  using 
one  to  keep  track  of  her  team.  Similarly,  the  Navy's  develop¬ 
ment  of  unmanned  surface  vessels  and  submarines  might 
become  boons  to  shipping  and  fishing. 

What  novice  or  unskilled  hunter  might  not  benefit  from  a 
highly  precise  scope,  or  better  yet,  smart  ammunition  on  the 
opening  day  of  deer  season?  What  about  improved  equip¬ 
ment,  including  energy  efficient  products,  for  campers?  Ex¬ 
amples,  and  potential,  abound. 

The  only  thing  now  standing  between  this  truly  modest  pro¬ 
posal  and  making  a  profit  on  defense  programs  is  the  will  of 
Congress  and  the  president.  And,  there  will  be  plenty  in  and 
out  of  Congress  who  will  argue  that  such  a  proposal  would 
harm  the  competitive  advantage  of  commercial  enterprises  in 
the  United  States.  However,  remember:  extremis  mail's  extrema 
remedia ,  what  we  have  come  to  use  as  "Desperate  times  call 
for  desperate  measures."  We  are  in  those  desperate  times, 
and  if  we  don't  do  something  about  it,  all  U.S.  businesses  will 
be  harmed. 

In  the  next  issue  of  Defense  AT&L,  we  fix  the  Social  Security 
funding  shortfall  problem.  & 

The  author  can  be  reached  atjohn.krieger@dau.mil. 
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On  Your  Way  to  the  To 
DAU  Can  Help  You  Get 

If  you're  in  the  Defense  Acquisition  Workforce,  you  need  to  know 
about  the  Defense  Acquisition  University.  Our  education  and 
training  programs  are  designed  to  meet  the  career-long 
training  needs  of  all  DoD  and  defense  industry  personnel. 


■  >; 


DAU  provides  a  full  range  of  basic,  intermedi¬ 
ate,  and  advanced  curricula  training,  as  well 
as  mission-specific  and  continuous  learn¬ 
ing  courses.  Whether  you're  new  to  the 
acquisition  workforce  or  a  seasoned 
member,  you  can  profit  from  DAU 
training. 

Convenient— Learn  where 
and  when  it  suits  you 


u 


DAU's  programs 
are  offered  at 
five  regional 
campuses 
and  their  addi¬ 
tional  training  sites. 

We  also  have  certification 
courses  taught  entirely  or  in 
part  through  distance  learning,  so 
you  can  take  courses  from  your  home 
or  office.  Check  out  the  100-plus  self-paced 
modules  on  our  Continuous  Learning  Center 
Website  at  http://clc.dau.mil. 

You'll  find  the  DAU  2012  Catalog  at  www.dau.mil.  Once 
you've  chosen  your  courses,  it's  quick  and  easy  to  register  on¬ 
line.  Or  contact  DAU  Student  Services  toll  free  at  888-284-4906  or 
student.services@dau.mil,  and  we'll  help  you  structure  an  educational 
program  to  meet  your  needs.  DAU  also  offers  fee-for-service  consulting  and 
research  programs. 


Comprehensive— Learn  what  you  need  to  know 


£AU 


Design  of  Experiments  for 
Information  Technology  Systems 


What  Program  Managers  Should  Know 
About  the  Plan  and  Design  Phases 


Rachel  T.  Silvestrini,  Ph.D.  ■  Maj.  William  J.  Parker  III  ■  Ginger  Sammito 


ecent  mandates  require  that 
rigorous  statistical  and  math¬ 
ematical  approaches  be  ap¬ 
plied  to  all  tests  that  fall  under 
developmental  and  operational 
test  and  evaluation  (T&E).  On 
October  19,  2010,  J.  Michael 
Gilmore,  director  of  Operational 
Test  and  Evaluation,  released  a 
memorandum  to  the  T&E  com¬ 
munity  within  the  DoD  that  de¬ 
scribes  an  initiative  designed 
to  increase  the  use  of  scientific 


Silvestrini  is  an  assistant  professor  in  the  Operations  Research  Department  at  the  Naval 
Postgraduate  School.  Parker  is  a  C4ISR  systems  operational  test  director  and  operations 
research  system  analyst  for  the  Homeland  Security/Information  Assurance  Portfolio  at  the 
Joint  Interoperability  Test  Command.  Sammito  is  a  principal  operations  research  system 
analyst  for  the  Force  Application/Force  Protection  Portfolio  at  the  Joint  Interoperability  Test 
Command. 
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experiment  is  planned  in  such  a  way  that 
minimizes  the  resources  spent,  while 
maximizing  the  information  obtained. 


and  statistical  methods  to  develop  rigorous  methods  for  test 
and  data  analysis.  Dr.  Gilmore's  memo  specifies  the  need  for 
using  rigorous  statistical  based  testing  methods  in  order  to  en¬ 
sure  that  proper  and  sufficient  data  is  collected  to  answer  the 
question  of  interest.  In  addition,  Edward  R.  Greer,  the  director 
of  Developmental  Test  and  Evaluation,  has  championed  the 
skillsets  of  design  of  experiments  (DoE),  statistics,  and  test  de¬ 
sign  principles  in  the  rejuvenation  and  development  of  the  T&E 
workforce  as  one  of  his  top  initiatives  to  the  practice  of  T&E. 

The  framework  that  encompasses  the  statistical  and  math¬ 
ematical  approaches  for  T&E  is  called  scientific  based  test 
design  (SBTD).  SBTD  can  be  applied  to  all  fields  and  applica¬ 
tion  areas  within  the  T&E  realm.  There  is  no  set  of  T&E  experi¬ 
ments  in  which  SBTD  does  not  apply.  For  example,  consider 
the  program  manager  (PM)  who  is  involved  with  IT  systems 
and  feels  that  SBTD  cannot  be  applied  to  his/her  respective 
system  because  the  variable  measures  of  interest  in  the  ex¬ 
periment  results  in  a  binary  outcome.  In  other  words,  did  the 
system  work  (yes  or  no)?  Although  this  is  a  formidable  chal¬ 
lenge  that  must  be  considered  prior  to  running  the  experiment, 
it  is  not  a  showstopper. 

SBTD  is  a  framework  that  includes  statistical  based  methods 
for  T&E  such  as  DoE  and  regression  analysis.  DoE  is  a  for¬ 
mal  approach  for  the  development  of  a  set  of  tests 
to  be  carried  out  in  an  experiment.  An  experiment  is 
a  large  number  of  individual  tests  (also  called  trials 
or  runs)  where  variables  are  manipulated  and  data 
is  collected. 

There  are  abundant  sources  of  literature  on  DoE  that 
describe  the  mathematical  and  statistical  based  tac¬ 
tics  for  designing  and  analyzing  the  results  of  an  ex¬ 
periment  that  can  meet  the  needs  of  any  experimen¬ 
tal  goals.  These  methods  ensure  that  valid,  objective, 
and  scientific  conclusions  are  reached.  Additionally, 
the  use  of  DoE  ensures  that  the  experiment  is  planned 
in  such  a  way  that  minimizes  the  resources  spent, 
while  maximizing  the  information  obtained.  Figure  1 
highlights  the  four  phases  of  the  DoE  approach:  Plan, 

Design,  Execute,  and  Analyze. 

Unlike  the  T&E  of  traditional  weapons  systems  such 
as  aircraft,  tanks,  artillery,  maritime  vessels,  etc., 
the  PM  involved  with  IT  systems  testing  may  expe¬ 


rience  slightly  different  challenges  associated  with  the  T&E 
processes.  However,  the  phases  of  DoE  process  do  not  change 
for  anyone.  While  this  article  is  primarily  aimed  at  the  PM 
within  T&E  of  IT  systems,  it  is  intended  to  be  beneficial  reading 
for  any  PM  involved  with  T&E  in  the  DoD.  The  remainder  of 
this  article  will  briefly  cover  how  to  apply  the  first  two  phases 
of  DoE  through  an  example  application  to  an  IT  system.  When 
appropriate,  specific  challenges  one  might  encounter  will  be 
highlighted. 

Applying  Science  Based  Testing  Designs 

The  DoE  approach  to  the  experiments  conducted  during  the 
T&E  process  is  displayed  in  Figure  1.  The  first  two  phases  of 
this  process  (Plan  and  Design)  will  be  discussed  through  an 
example  application  to  an  IT  system. 

Suppose  that  a  PM  is  in  charge  of  oversight  for  a  new  soft¬ 
ware  application  being  developed  as  a  test  tool.  The  experi¬ 
ment  used  to  test  the  software  is  called  Bravo  Test.  During 
Bravo  Test  different  message  types  for  multiple  platforms 
with  an  Identification  Friend  or  Foe  (IFF)  system  are  both 
transmitted  and  received.  A  DoD  architecture  framework 
is  illustrated  in  Figure  2.  Bravo  Test  will  take  place  at  the 
systems  level  (middle  view). 


Figure  1.  Design  of  Experiments  (DoE)  Process 
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Phase  1:  Plan 

The  first  phase  in  the  DoE  process  is  Plan.  This  phase  includes 
statement  of  the  goal  of  the  experiment  as  well  as  the  develop¬ 
ment  of  a  list  of  variables  involved  in  the  experiment.  There 
are  three  types  of  variables  important  to  list: 

■  variables  that  will  be  manipulated  or  controlled  during  the 
experiment 

■  variables  that  cannot  be  controlled,  but  may  change  dur¬ 
ing  the  experiment 

■  variables  used  to  measure  the  system  (outcomes) 

The  goal  of  Bravo  Test  is  to  test  the  accuracy  and  timeliness 
of  messages  transmitted  and  received.  The  first  objective  of 
Bravo  Test  is  to  determine  whether  or  not  each  of  four  differ¬ 
ent  platforms  transmits  or  receives  messages  with  accuracy 
rate  above  99  percent.  The  second  objective  is  to  model  the 
expected  time  to  transmit  and  receive  a  message  as  a  func¬ 
tion  of  the  different  platforms,  identification  systems,  and  type 
of  message.  The  PM  should  be  aware  that  the  recognition  of 
the  goal  and  objectives  in  a  test  often  aid  in  identifying  the 
variables  present  in  the  experiment. 

Table  1  illustrates  the  three  different  controllable  variables  that 
will  be  manipulated  (changed)  over  the  course  of  Bravo  Test. 
Remember;  variables  that  can  be  controlled  as  well  as  those 
that  cannot  be  controlled  should  be  identified.  For  example, 
during  Bravo  Test  the  average  system  load  during  the  trans¬ 
mission  of  a  message  may  be  measurable,  but  it  may  not  be  a 
variable  that  is  directly  controllable.  The  PM  should  be  eager 
to  identify  all  uncontrollable  variables  possible  and  additionally 
keep  in  mind  that  it  is  possible  that  a  few  variables  may  not 
be  known  initially,  but  will  emerge  later.  This  should  not  be  a 


stumbling  point,  but  an  opportunity  for  the  PM  to  refine  the 
test  during  the  next  cycle  with  more  information.  This  involves 
going  back  to  the  planning  phase  and  proceeding  from  there. 

Example  Factors  to  be  varied 
during  Bravo  Test 


Controllable  Variables 

Settings  During  Test 

IFF  (Identification,  Friend, 
or  Foe) 

Range  0  -  5 

Message  types 

UTF-8,  UTF-16,  UTF-32  (UTE  = 
Unicode  Transformation  Format) 

Producing  or  Consuming 
Platforms 

A,  B,  C,  D 

In  Bravo  Test,  there  are  two  outcome  variables:  (1)  accuracy 
of  message  and  (2)  time  to  transmit/receive  message.  Ac¬ 
curacy  is  a  binary  variable:  if  the  message  is  100  percent  cor¬ 
rect,  the  data  point  will  be  considered  1  (accurate);  otherwise 
0  (not  accurate).  In  IT  systems  testing,  a  binary  response  is 
a  common  metric  of  interest.  Also,  many  outcome  variables 
may  be  collected  for  a  single  test  within  the  experiment;  this 
is  important  to  note  and  is  used  when  assessing  the  quantity 
of  tests  required  for  the  experiment. 

Without  proper  care  in  the  Plan  phase  of  the  experiment,  the 
direction  of  the  experiment  may  become  unclear.  This  leads  to 
the  collection  of  erroneous  or  incomplete  information,  which 
will  prevent  the  experimental  goals  from  being  met.  Often, 
determining  the  variables  of  interest  in  an  experiment  can  be 
a  difficult  task  that  should  be  undertaken  with  caution.  Fish¬ 
bone  diagrams  as  well  as  other  brainstorming  techniques  often 
work  well  during  subject  matter  expert  meetings  to  discuss 
variable  selection. 


Figure  2.  DoD  Architecture  Framework 
with  Systems  View  in  Center 


The  Operational  View  describes 
and  interrelates  the  operational 
elements,  tasks  and  activities,  and 
information  flows  required  to 
accomplish  mission  operations. 


The  Systems  View  describes  and 
interrelates  the  existing  or  postu¬ 
lated  technologies,  systems,  and 
other  resources  intended  to  support 
the  operational  requirements. 


The  Technical  View  describes  the 
profile  of  rules,  standards,  and 
conventions  governing  systems 
implementation  and  forecasts  their 
future  direction. 


DoD  Architectural  Framework  (DoDAF) 


Phase  2:  Design 

The  Design  Phase  involves  map¬ 
ping  out  the  sets  of  tests  that  will 
be  conducted  during  the  experi¬ 
ment.  Specifically,  this  phase  in¬ 
volves  the  selection  of  the  design 
type  and  the  determination  of  the 
number  of  tests  to  be  conducted 
in  the  experiment  (also  known  as 
sample  size).  Each  test  involves 
the  control  and  manipulation  of 
variables  identified  in  the  Plan 
Phase.  There  are  a  number  of 
different  experimental  design 
techniques  found  in  various  text¬ 
books,  journal  articles,  technical 
reports,  and  case  studies. 

Examples  of  design  selections 
include  factorial  design,  frac¬ 
tional  factorial  design,  central 
composite  design,  covering 
array,  and  optimal  design.  While 
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Figure  3.  JMP — User  Interlace  for  the 
Development  of  Full  Factorial  Design 
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a  PM  does  not  necessarily  need  to  know 
each  different  design,  they  should  recognize 
that  different  designs  are  appropriate  for 
different  experimental  goals.  For  example,  a 
fractional  factorial  design  is  an  appropriate 
design  choice  when  the  experimental  goal 
involves  finding  the  subset  of  factors  that 
influence  the  outcome  variable  of  interest. 
This  is  a  goal  typically  encountered  in  the 
early  phase  of  testing.  For  situations  involv¬ 
ing  multiple  responses  with  overlapping  or 
conflicting  goals,  a  hybrid  design  approach, 
in  which  different  design  choices  are  com¬ 
bined,  can  be  used  to  satisfy  all  objectives 
of  the  experiment. 

In  addition  to  design  choice,  the  number  of 
tests  to  run  (or  the  sample  size)  of  an  experi¬ 
ment  must  be  determined  duringthis  phase. 
Given  the  opportunity,  a  PM  might  prefer  to 
choose  an  unlimited  sample  size.  However, 
cost,  time,  and  resource  constraints  often 
drive  sample  size  choices. 


For  Bravo  Test,  a  full  factorial  design  with 
four  replicates  is  selected  to  support  the 
goals  of  testing  the  accuracy  and  timeliness 


Figure  4.  JMP — Full  Factorial  Table  Design 


of  messages  transmitted  and  received.  A 
statistical  software  package,  such  as  JMP 
(illustrated),  can  be  used  to  create  the  de¬ 
sign.  Snapshots  of  the  design  creation  are 
shown  in  Figure  3  and  Figure  4.  Figure  3  il¬ 
lustrates  the  user  interface  that  guides  the 
inputs  to  the  development  of  the  design. 
Figure  4  contains  the  design.  The  design 
dictates  the  running  of  every  experimental 
test.  For  example,  the  first  experimental 
test  will  be  conducted  with  IFF  =  2,  Mes¬ 
sage  Type  =  UTF-16,  and  Platform  =  D. 

A  full  factorial  design  is  appropriate  for 
the  needs  of  Bravo  Test.  In  Bravo  Test, 
simple  relationships  between  IFF,  Mes¬ 
sage  Type  and  Platform  will  be  inves¬ 
tigated.  In  other  situations,  different 
designs  may  be  more  apt.  The  factorial 
design  dictates  a  baseline  number  of  runs 
in  the  experiment.  That  number  can  be 
altered  by  repetition  of  the  experiment 
(as  seen  in  one  of  the  selection  tabs  in 
Figure  3).  It  is  important  for  the  PM  to 
realize  that  within  a  resource-constrained 
environment,  a  single  experiment  cannot 
provide  unlimited  answers.  Both  design 
choice  and  sample  size  restrictions  trans¬ 
late  to  restrictions  on  what  information 
can  be  obtained.  Statistical  and  math- 
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ematical  analysis  can  greatly  help  overcome  sample  size 
dilemma  by  focusing  on  answering  the  following: 

■  Given  a  fixed  sample  size,  what  information  can  be  mea¬ 
sured  and  modeled? 

■  Given  measurement  or  modeling  requirements,  what 
sample  size  is  required? 

Approach  (1)  involves  identifying  risks  in  the  constrained  envi¬ 
ronment  and  approach  (2)  involves  determining  requirements 
of  sample  size  based  on  the  risks  the  experimenter  is  will¬ 
ing  to  accept.  Risks  can  be  discussed  in  terms  of  confidence 
level  and/or  power  of  mathematical  estimation.  These  are 
two  terms  related  to  statistical  analysis  that  PMs  should  be 
or  become  familiar  with. 

During  the  Design  Phase,  the  PM  should  encourage  documen¬ 
tation  of  the  methodology  that  includes  rationale  for  selecting 
a  design,  sample  size,  and  lessons  learned  from  the  process. 
Clear  documentation  will  help  the  PM  face  the  challenges  of 
the  iterative  DoE  process  and  development  stages  as  the  soft¬ 
ware  moves  towards  maturity. 

Conclusion 

SBTD  methods,  specifically  DoE,  can  and  should  be  applied 
to  T&E  of  IT  systems.  There  are  many  case  studies  that  docu¬ 
ment  the  success  of  the  DoE  approach  for  both  IT  and  non- 
IT  systems.  This  article  covered  the  Plan  and  Design  phases 
in  the  DoE  approach.  It  is  believed  that  the  Plan  and  Design 


phases  are  of  utmost  importance  because  an  inadequately 
designed  experiment  will  result  in  poor  results  and  possibly 
incorrect  conclusions,  thus  making  the  Execute  and  Analyze 
phases  meaningless. 

The  Execute  Phase  refers  to  the  running  of  each  test  in  the 
experiment.  For  Bravo  Test,  the  experiment  to  be  run  is  illus¬ 
trated  in  Figure  4.  During  this  phase,  it  is  imperative  that  each 
test  is  run  to  specification.  This  involves  ensuring  that  proper 
blocking,  randomization,  and  replication  are  carried  out  as 
specified  by  the  design.  The  Analyze  Phase  encompasses  a 
mathematical  study  of  the  resulting  data  to  obtain  valid  and 
objective  conclusions. 

Sometimes  the  challenges  and  decisions  in  the  creation  of 
an  experimental  design  approach  appear  endless  for  the  PM, 
especially  as  requirements  shift  from  traditional  testing  to  rig¬ 
orous  SBTD  for  IT  systems.  The  PM  must  ensure  compliance 
with  applicable  policies.  The  PM  is  also  responsible  for  the 
quality  and  consistency  to  those  standards  while  developing 
test  reports  based  on  a  sound,  scientific  rigor  that  have  not 
formally  been  a  part  of  any  IT  system/program.  The  PM  needs 
to  look  beyond  the  present  in  facing  these  SBTD  challenges 
in  IT  systems  and  focus  on  the  valid,  objective,  and  measure- 
able  approach  that  ultimately  saves  time  and  money  over  the 
development  cycle  of  the  IT  system.  & 

The  authors  can  be  reached  at  rtsilves@nps.edu,  william.j.parker60.mil@ 
mail.mil,  and  ginger.j.sammito.civ@mail.mil. 
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Denise  A.  Little 


We've  outlined  my  program's 
command  [cost]  requirements 
consistently  and  in  detail  over 
the  past  3  years.  I  don't  want 
to  waste  time  on  a  semantics 
debate  over  what  I  call  my 
requirements. 

—Program  manager  during 
a  weapon  system  review 


We  could  all  learn  a  thing  or  two  from 
the  Oakland  A's  general  manager, 
Billy  Beane,  who  took  his  team  with 
a  shoestring  budget  to  the  play-offs 
in  2002.  The  A's  spent  $41  million 
on  player  salaries,  compared  with  the  New  York  Yan¬ 
kees,  who  spent  more  than  $125  million.  According 
to  Michael  Lewis'  Moneyball:  The  Art  of  Winning  an 
Unfair  Game,  Billy's  plan  used  performance  metrics 
to  select  players,  recruit  an  entire  team,  and  com¬ 
pete  against  teams  like  the  Yankees,  who  can  afford 
practically  any  player  they  want. 


Little  is  the  operations  officer  for  the  Marine  Corps  Logistics  Vision  &  Strategy  Branch  at  Headquarters  Marine  Corps.  She  is  a  retired  Army  Aviation 
Logistician  and  Maintenance  Officer r  an  acquisition  logistician ,  and  a  sustainment  cost  analyst.  She  was  a  member  of  the  Army  G-4's  Army  Campaign 
Plan  2012  writing  team. 
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Billy  successfully  played  hardball  when  it  came  to  recruiting 
and  trading  yet  took  care  of  the  players,  the  organization,  and 
the  fans  with  a  poetic  integrity.  He  committed  to  his  strategic 
vision  and  tactical  plan.  He  performed  analysis,  coached  fi¬ 
nancial  stakeholders  and  peers  alike,  and  he  even  taught  those 
who  doubted  the  process.  He  went  the  distance— and  years— 
to  convince  them  of  the  hard  right  over  the  easy  wrong.  He 
realized  that  deliberate  planning  and  analytics  take  time  and 
patience.  He  is  a  change  agent  in  Major  League  Baseball's  own 
brand  of  cost  culture.  So  how  do  Cost  Warriors  apply  Billy's 
winning  approach  to  strategic  communications? 

Overwhelming  Underdogs 

Take  a  look  at  the  role  that  the  Army  G-4  plays  in  cost  man¬ 
agement,  and  compare  it  to  Billy's  strategic  plan.  The  Army 
G-4  logistician  analyzes  sustainment  costs  to  provide  deci¬ 
sion  support  for  the  secretary  of  the  Army  and  the  assistant 
secretary  of  the  Army  for  acquisition,  logistics  and  technology 
(ASA[ALT])  concerning  system  life  cycle  cost  or  total  own¬ 
ership  cost.  (The  ASA[ALT]  is  also  the  army  acquisition  ex¬ 
ecutive.)  Acquisition  executives  must  consider  both  cost  and 
affordability  when  approving  milestone  decisions.  Sustainment 
costs  include  depot-level  maintenance,  software  maintenance, 
logistics  assistance  representatives,  and  more. 

To  give  you  perspective  on  the  size  of  the  budget  that  the  Army 
G-4  manages,  it  averages  4  percent  of  Army  resources.  Out 
of  six  Army  Program  Evaluation  Groups  (PEGs),  Army  G-4's 
Sustaining  PEG  comes  in  next  to  last  place,  just  ahead  of  the 
Organizing  PEG.  According  to  recent  G-8  Program  Analysis  & 
Evaluation  (PAE)  office's  figures,  manpower  annually  leads  the 
pennant  race  with  a  whopping  40-45  percent  of  the  budget. 
Consider  that  procurement  comes  in  a  distant  second  between 
14  percent  and  18  percent.  However,  the  manpower  portion 
of  the  budget  doesn't  include  personnel  labor  for  those  who 
perform  depot  and  other  sustainment.  They  include  Field 
Software  Engineers  who  perform  software  support,  logistics 
assistance  representatives  who  provide  technical  guidance, 
and  engineers  who  provide  modification  work  order  support. 

I  was  reminded  by  Cecile  Batchelor,  the  Army  cost  culture 
initiative  program  manager  and  special  assistant  to  the  deputy 
assistant  secretary  of  the  Army  for  cost  and  economics,  that 
we  should  compare  historical  data  to  our  long-term  budget  ex¬ 
pectations.  "We  should  ask  ourselves  how  our  original  expec¬ 
tations  compare  to  our  current  budget  requests,  and  ask  what 
we  are  doing  to  improve.''  Marc  Gutleber,  of  the  U.S.  Army 
Communications-Electronics  Command  (CECOM),  led  Army 
G-4  cost  analysts  with  building  the  software  maintenance/ 
sustainment  cost  element  structure  and  each  element's  defini¬ 
tions.  He  added  to  Batchelor's  questions  on  budget  expecta¬ 
tions.  "We  should  also  ask  what  changed  since  the  original 
expectation.  Did  the  projected  'cost  savings'  ever  materialize? 
Did  the  sustainment  costs  dramatically  increase,  and  why?" 
He  noted  that  the  Army  had  planned  to  field  a  system  at  divi¬ 
sion  headquarters  level,  then  changed  the  plan  to  field  it  down 
to  the  brigade  level;  this  increased  the  necessary  number  of 


systems  by  the  hundreds.  The  result  was  an  apples-to-oranges 
comparison  of  the  original  estimate  produced  at  milestone  A 
to  the  estimate  produced  years  later  for  the  full  rate  produc¬ 
tion  milestone  review. 

So  why  go  to  all  this  trouble  to  apply  cost  management  to 
the  sustainment  programming  and  budget  process?  There's 
a  DoD-wide  problem  of  low-balling  sustainment  costs,  and  in 
our  constrained  budget  environment  we  need  to  have  realistic 
estimates  to  understand  what  we're  committing  ourselves  to 
over  the  20-  to  30-year  system  life.  Here's  a  notional  example 
of  software  maintenance.  At  milestone  A,  a  program  manager 
estimates  software  maintenance  at  $1  billion  per  year  in  the 
original  life  cycle  cost.  Then  by  milestone  C,  the  estimate  bal¬ 
looned  to  $6  billion.  Less  than  a  year  later  during  the  next  pro¬ 
gram  objective  memorandum  (POM)  build,  it  increased  again 
to  $7  billion.  How  did  this  happen?  To  make  a  long  story  short: 
each  agency  and  each  forum  had  a  different  way  of  defining 
software  maintenance.  Each  had  its  own  set  of  cost  elements 
with  different  definitions  for  each  cost  element. 

The  low  sustainment  estimates  are  further  complicated  by  the 
annual  competition  for  a  piece  of  the  operations  and  main¬ 
tenance  (O&M)  appropriations  budget  pie.  The  life  cycle 
management  commands  (LCMC)  that  fall  under  the  Army 
Materiel  Command  (AMC)  present  sustainment  command 
requirements  that  are  consolidated  into  functions  that  sup¬ 
port  multiple  systems.  Sustainment  command  requirements 
are  funded  through  functional  channels,  such  as  sustainment 
systems  technical  support  (SSTS)  programming  that  includes 
logistics  assistance  representatives,  second  destination  trans¬ 
portation,  and  post  production  software  support  (PPSS).  They 
are  not  always  specific  to  the  individual  program.  The  program 
manager  as  the  total  life  cycle  manager  partners  with  his/ 
her  LCMC(s),  the  Army  G-4,  G-3,  and  G-1,  in  the  projection, 
management  and  synchronization  of  the  O&M  appropriations. 

Say  it  ain’t  so,  Joe! 

The  Army  G-4  came  up  with  its  strategic  game  plan  for 
cost  management  implementation  in  Fiscal  Year  (FY)  2010. 
Chaired  by  Joint  Staff's  director  of  logistics  (J-4)  and  facilitated 
by  the  deputy  assistant  secretary  of  Defense  for  program  sup¬ 
port,  the  Defense  Materiel  Readiness  Board  (DMRB)  tasked 
each  Service's  Deputy  Chief/Commandant  for  Logistics  to 
outline  an  approach  to  total  ownership  cost  optimization. 

It  was  Oakland's  (Army  G-4's)  turn  on  the  pitching  mound  to 
brief  the  DMRB.  Their  integrity  was  on  the  line.  Did  the  coach 
send  its  Cy  Young  contender  to  the  mound  with  instructions  to 
intentionally  walk  the  batter?  Gloss  over  existing  Army  regula¬ 
tions  and  say  that  the  World  Series  POM  FY  2013-2017  cham¬ 
pionship  was  all  cinched  up? 

No,  the  team  managers  took  the  hard  right.  The  pitcher  aimed 
for  the  board's  strike  zone  to  deliver  a  meaningful  discussion 
on  their  current  status,  even  though  it  meant  extra  innings  and 
an  extended  season. 
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It  ain’t  over  till  it’s  over. 

What  was  the  Army's  status?  There  are  regulations  and  di¬ 
rectives— lots  of  'em!  But  how  well  were  they  implemented? 
How  clearly  do  we  understand  them?  Some  were  ambiguous 
or  at  times  in  conflict  with  others.  And  saying  so  would  mean 
sending  some  starting  pitchers  to  the  dugout  during  the  fourth 
inning.  But  despite  challenges,  they  had  a  plan  and  were  al¬ 
ready  making  steady  progress. 

They  reviewed  existing  standards  and  identified  a  few  that 
were  unrealistic.  They  socialized  the  plan  to  gain  momentum 
using  existing  forums,  such  as  the  Cost  Review  Board  working 
group  and  the  Weapon  System  Reviews.  They  weren't  rein¬ 
venting  the  wheel,  just  making  it  better.  They  set  out  to  make 
sustainment  cost  estimates  reliable,  repeatable,  and  accurate. 


ing,  fielding,  and  equipping  actions)  pays  for  post  deployment 
software  support  (PDSS),  through  the  Equiping  PEG. 

PDSS  is  like  hundreds  of  pop-up  balls  flying  into  the  bleachers 
during  batting  practice.  In  other  words,  they  are  a  multitude  of 
software  issues  that  require  patches  and  troubleshooting,  in 
addition  to  the  operational  maintenance  costs  such  as  paying 
for  licenses,  implementing  information  assurance  vulnerabil¬ 
ity  assessments,  certification,  providing  field  level  software 
maintenance,  etc.  If  software  maintenance  occurs  during  or 
after  the  first  full  year  after  the  hardware  production  line  ends, 
software  maintenance  typically  is  in  its  steady  state.  This  is 
post  production  software  support  (PPSS).  If  it's  PPSS  and  or¬ 
ganically  supported,  then  the  Army  G-4  pays  for  it. 


To  give  you  perspective  on  the  size 
of  the  budget  that  the  Army  G-4 
manages,  it  averages  4  percent  of 
Army  resources. 


You  can  observe  a  lot  by  watching. 

The  Army  G-4  talked  with  product  support,  business  and  cost 
managers,  and  explained  to  them  that  the  Army  was  in  new 
territory.  The  three  managers  make  up  the  vital  team  who 
would  translate  their  logistics/sustainment  support  jargon 
into  the  appropriate  sustainment  cost  elements. 

Then  we  throw  into  the  semantics  batting  cage  the  various 
ways  to  use  the  word  "sustainment.”  When  we  hear  the  word 
sustainment  and  find  out  that  there's  a  Sustaining  PEG,  we 
might  assume  that  all  sustainment-related  costs  are  paid  by 
the  Army  G-4  Logistician.  But  the  Army  G-3  Operations  and 
Training  team  pays  for  spare  and  repair  part  replenishment, 
for  field-level  maintenance  and  other  expenses,  through  the 
Training  PEG. 

Programming  for  software  maintenance  can  also  be  confusing. 
It  contains  the  word  "maintenance,”  so  naturally  the  Army  G-4 
pays  for  maintenance,  right?  It  depends  on  when  the  main¬ 
tenance  occurs.  The  transition  occurs  during  the  first  full 
year  after  the  hardware  production  line  ends.  If  the  software 
maintenance  occurs  before  the  first  full  year  after  hardware 
production  line  ends,  then  the  Army  G-8  (responsible  for  fund- 


There's  one  more  software  maintenance  twist!  Does  the  soft¬ 
ware  maintenance  support  a  business  or  software-intensive 
system  that's  not  tied  to  a  weapon  system  or  its  hardware?  If 
so,  the  primary  stakeholder  is  the  post  production  software 
maintenance  bill  payer.  The  Integrated  Personnel  and  Pay  Sys¬ 
tem-Army  (IPPS-A)  business  system  illustrates  this  point.  It's 
a  personnel  business  system,  so  the  Army  G-1  is  the  primary 
stakeholder  and  is  responsible  for  programming  and  managing 
those  post  production  software  maintenance  funds.  The  same 
is  true  of  the  Army  G-4  who  pays  for  the  Logistics  Manage¬ 
ment  Program  business  system's  post  production  software 
maintenance. 

Sweet  Caroline,  good  times  never  seemed 
so  good. 

While  we  need  to  know  how  much  the  program  costs,  whose 
office  decides  if  it's  affordable?  Who  decides  who  will  program 
the  money?  That's  Army  staff  business  in  the  franchise  head 
shed,  right?  The  truth  is  that  we  need  to  know  its  cost,  afford¬ 
ability,  and  stakeholders  to  ensure  that  we  programmed  and 
budgeted  for  it  properly.  So  let's  go  to  the  Cost  Warrior  basics 
for  the  plan. 
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First,  if  we  know  how  much  it  costs,  then  we  can  figure  out  if 
we  can  afford  it.  We  must  initially  determine  the  true  cost  by 
defining  each  cost  category,  that  is,  the  cost  element  structure. 
The  program  office  estimate  (POE)  cost  categories  and  their 
definitions  must  match  those  used  by  the  budget,  cost,  and 
logistics  managers  throughout  the  Army. 

Second,  is  it  affordable?  Do  we  have  enough  money  budgeted 
over  the  next  five  years  to  cover  our  costs?  More  importantly 
do  we  have  enough  to  cover  costs  over  the  system's  entire 
life  (total  ownership  cost  or  life  cycle  cost)?  Did  we  consider 
just  the  one  system?  Or  did  we  consider  all  the  systems  that 
make  up  one  portfolio  consisting  of  similar  systems  that  have 
a  related  or  parallel  mission?  In  an  address  to  an  Armor  War¬ 
fighting  Conference,  Gen.  Peter  Chiarelli  said,  "If  you  look  at 
any  one  of  these  systems  as  an  individual  system,  you  can 
sell  just  about  anything.  But,  when  you  look  at  the  entire 


(O&M)  appropriation's  [Sustaining  PEG]  Total  Obligation  Au¬ 
thority  (TOA)  decreased  in  FY12,  and  will  continue  to  decrease 
over  the  next  five  years  as  the  Army  implements  the  SECDEF's 
[Secretary  of  Defense]  efficiencies  initiatives." 

—Army  Campaign  Plan  2012 

Let’s  root,  root,  root  for  the  home  team. 

The  Army  Materiel  Enterprise  team  will  have  confidence  in 
their  estimates,  because  everyone  will  be  using  the  same 
cost  category  definitions.  The  O&M  sustainment  estimate 
becomes  reliable,  repeatable,  and  accurate.  Let's  use  analyti¬ 
cal  gauges  of  cost  estimates  to  enable  ourselves  to  compete 
successfully  during  the  POM  validation  process. 

Applying  cost  analysis  to  a  program  develops  your  cost-man¬ 
agement  and  cost-analysis  skills  and  makes  you  competent  in 
this  emerging  skill  set.  You  can  use  cost  analysis  to  determine 


There's  a  DoD-wide  problem  of  low-balling 
sustainment  costs,  and  in  our  constrained 
budget  environment  we  need  to  have 
realistic  estimates  to  understand  what  we're 
committing  ourselves  to  over  the  20-  to  30- 
year  system  life. 


portfolio  you  can  start  to  see  where  we  have  duplication  in 
different  systems  or  maybe  we're  overinvesting  in  one  and 
underinvesting  in  another."  Perhaps  we  could  incorporate  a 
20-30  year  strategic  planning  budget  that  includes  all  sus¬ 
tainment  costs. 

The  future  ain’t  what  it  used  to  be. 

Third,  change  takes  time  and  patience.  Over  the  past  10  years, 
big  money  has  been  flowing.  It  was  as  if  the  San  Francisco 
Giants  starting  pitcher,  Tim  Lincecum,  brought  on  the  heat 
with  incessant  fast  balls  and  change-ups  in  the  form  of  never- 
ending  deployments  and  high  operational  tempo.  If  we  needed 
more  than  the  anticipated  base  dollars,  we  would  go  to  his 
all-star  teammate  and  closing  pitcher,  Brian  Wilson— also 
known  as  Overseas  Contingency  Operations  (OCO)  budget 
requests.  But  the  future  is  now,  and  we  find  ourselves  with 
more  fiscal  constraints  than  ever.  Something  has  to  give  as 
base  dollars  decrease  and  continual  OCO  dollars  become  a 
distant  memory. 

"The  Army's  ability  to  balance  its  fiscal,  materiel,  and  personnel 

requirements  is  critical  since  the  Operations  &  Maintenance 


player  performance  and  come  up  with  a  winning  strategy  for 
the  entire  portfolio.  How  are  we  doing  compared  to  the  original 
expectation  (our  estimate)?  If  not  great,  what  are  we  doing  to 
improve?  Managing  sustainment  life  cycle  costs  means  an¬ 
swering  those  questions  to  snag  a  spot  on  the  Cost  Warrior 
team. 

Each  agency  could  set  a  goal  to  win  the  cost  management 
Triple  Crown.  What  if  you  prepared  action  officers  to  become 
your  agency's  Los  Angeles  Dodger  Clayton  Kershaw,  who  won 
the  Triple  Crown  in  2011?  First,  encourage  them  to  sharpen 
cost  management  and  analysis  skills  by  taking  Defense  Acqui¬ 
sition  University  online  and  resident  courses.  Second,  ask  them 
to  attend  the  four-hour  cost-benefit  analysis  course.  Third,  set 
their  sights  on  the  four-week  resident  cost-management  cer¬ 
tificate  course  at  the  Naval  Postgraduate  School  in  Monterey, 
Calif.  The  Triple  Crown?  Hey,  it  could  happen!  & 

The  author  can  be  reached  at  denise.little1@us.army.mil. 
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